[circt][arc] Add bufferized array handling (#10391)

* [circt][arc] Add bufferized array handling

Previously, !hw.array types were lowered directly to !llvm.array types. This
is the obvious lowering, but unfortunately it produces very poor code and can
in some cases bog down LLVM quadratically.

LLVM's ArrayType is aggressively SROA'd - it is really assumed to be more like
a tuple-type - so no matter how we try to avoid it (e.g. HWSpillCache), LLVM
will expand each array into individual scalars. This creates a large number of
instructions, but worse - it creates very difficult scheduling problems with
high connectivity. LLVM's pre-regalloc scheduler has been seen to take hours
on a problem that should be fairly simple.

This PR fixes this issue by introducing an !arc.arrayref<T> type. This mirrors
!hw.array<T>, except that it has buffer semantics rather than value semantics.
That is, !arc.arrayref is to memref as !hw.array is to tensor.

I did heavily look into using the exising bufferization passes in MLIR, but
these are (a) quite hard to use, and (b) heavily tailored to the "tensor" and
"memref" types.

In the end it felt like much less complexity to implement our own barebones
bufferizer.

Each !hw.array op is translatable to an !arrayref op, except that the arrayref
versions take a destination buffer to modify as an operand. They are mostly in
destination passing style.

The translation to LLVMIR dialect is trivial (!arrayref -> !llvm.ptr).

There are other optimizations we can do to reduce the compile time (and runtime)
of array-heavy code but they are left for further PRs:

  * Aggressive reduction of arc.arrayref.alloc ops (better bufferization).
  * Optimization of hw.array_creates where array permutations/shuffles have been
    expanded.
  * Proper handling of nested arrays.

* Add dereferenceable(N) attribute

* Apply review comments:
  * Inherit ArrayRefType from ShapedType
  * Attribute:$size -> uint64_t:$size
  * Add traits and predicates to all ops for better verification
  * Add missing rewriter.modifyOpInPlace
  * Fix erroeous UpperCamelCasing

* Add support for nested arrays and arrays within structs.

We support these by never converting a nested array, or any struct containing an array.

Only "raw" arrays are converted to ArrayRefs.

Introduces `arc.arrayref.from_array` and `arc.arrayref.to_array` as casters because
hw.array is now expected to be legal in some scenarios.

* Switch on by default! (yay!)

* Fix integration test:
  * Explicitly declare Struct*Op as legal.
  * Plant a local copy of the array in StateReadOp.

Assisted-by: Gemini:experimental to identify the StateReadOp bug.

* Review comments:
  * DenseI64ElementsAttr -> ArrayAttr (again)
  * Handle OOB load, store and slice.
  * Remove extra alloca in slice lowering.
* Replace Load/Store with destructured insertvalue/extractvalue
* Act on review comments
* Add type verifier to disallow !arc.arrayref<0xT>

---------

Co-authored-by: James Molloy <jmolloy@google.com>

Author: jmolloy

include/circt/Dialect/Arc/ArcOps.td

@@ -551,7 +551,8 @@ def RootOutputOp : ArcOp<"root_output", [
 // Storage Access
 //===----------------------------------------------------------------------===//
 
-def AllocatableType : AnyTypeOf<[StateType, MemoryType, StorageType]>;
+def AllocatableType : AnyTypeOf<[StateType, MemoryType, StorageType,
+                                 ArrayRefType]>;
 
 def StorageGetOp : ArcOp<"storage.get", [Pure]> {
   let summary = "Access an allocated state, memory, or storage slice";
@@ -1134,4 +1135,146 @@ def RuntimeModelOp : ArcOp<"runtime.model", [
   }];
 }
 
+//===----------------------------------------------------------------------===//
+// ArrayRef ops
+//===----------------------------------------------------------------------===//
+//
+// Ops that deal with bufferized arrays. The !hw.array type has value semantics;
+// the !arc.arrayref type is the equivalent but with reference semantics.
+
+def ArrayRefAllocOp : ArcOp<"arrayref.alloc", []> {
+  let summary = "array allocation";
+  let description = [{
+    Allocates an array on the stack. If `init` is set, the array is initialized
+    to this value. Otherwise the array's contents are undefined.
+  }];
+  let arguments = (ins OptionalAttr<ArrayAttr>:$init);
+  let results = (outs ArrayRefType:$output);
+  let assemblyFormat = "attr-dict (`init` `(` $init^ `)`)? `:` type($output)";
+  let hasVerifier = 1;
+}
+
+def ArrayRefGetOp : ArcOp<"arrayref.get", [
+  MemoryEffects<[MemRead]>,
+  TypesMatchWith<"array element type matches the result type", "input", "value",
+                 "cast<ArrayRefType>($_self).getElementType()">
+]> {
+  let summary = "array get";
+  let description = [{
+    Loads a single value from an arrayref.
+  }];
+  let arguments = (ins ArrayRefType:$input, Index:$index);
+  let results = (outs AnyType:$value);
+  let assemblyFormat = [{
+    $input `[` $index `]` attr-dict `:` type($input) `->` type($value)
+  }];
+}
+
+def ArrayRefInjectOp : ArcOp<"arrayref.inject", [
+  MemoryEffects<[MemRead, MemWrite]>,
+  AllTypesMatch<["input", "output"]>,
+  TypesMatchWith<"element type matches input element type", "input", "element",
+                 "cast<ArrayRefType>($_self).getElementType()">
+]> {
+  let summary = "array inject";
+  let description = [{
+    Stores a single value into an arrayref.
+  }];
+  let arguments = (ins ArrayRefType:$input, Index:$index, AnyType:$element);
+  let results = (outs ArrayRefType:$output);
+  let assemblyFormat = [{
+    $input `[` $index `]` `,` $element attr-dict `:` type($input) `,` type($element) `->` type($output)
+  }];
+}
+
+def ArrayRefSliceOp : ArcOp<"arrayref.slice", [
+  Pure,
+  AllElementTypesMatch<["input", "output"]>
+]> {
+  let summary = "array slice";
+  let description = [{
+    Slices an array into a sub-array, returning a view into the original array.
+  }];
+  let arguments = (ins ArrayRefType:$input, Index:$lowIndex);
+  let results = (outs ArrayRefType:$output);
+  let assemblyFormat = [{
+    $input`[`$lowIndex`]` attr-dict `:` functional-type($input, $output)
+  }];
+}
+
+def ArrayRefCopyOp : ArcOp<"arrayref.copy", [
+  MemoryEffects<[MemRead, MemWrite]>,
+  SameOperandsAndResultType,
+]> {
+  let summary = "array copy";
+  let description = [{
+    Copies the content of one array into another array. Returns the input array.
+  }];
+  let arguments = (ins ArrayRefType:$input, ArrayRefType:$source);
+  let results = (outs ArrayRefType:$output);
+  let assemblyFormat = [{
+    $input `=` $source attr-dict `:` type($input)
+  }];
+}
+
+def ArrayRefCreateOp : ArcOp<"arrayref.create", [
+  MemoryEffects<[MemWrite]>,
+  AllTypesMatch<["input", "output"]>,
+  TypesMatchWith<"operand types match element type",
+                 "output", "elements", "SmallVector<Type, 2>("
+                 "cast<ArrayRefType>($_self).getNumElements(), "
+                 "cast<ArrayRefType>($_self).getElementType())">
+]> {
+  let summary = "array create";
+  let description = [{
+    Creates an array from a list of values. The values are ordered from
+    most significant to least significant (the same as `hw.array_create`).
+
+    The `input` array is populated with `elements` and is returned.
+  }];
+  let arguments = (ins ArrayRefType:$input, Variadic<AnyType>:$elements);
+  let results = (outs ArrayRefType:$output);
+  let assemblyFormat = [{
+    $input `=` $elements attr-dict `:` type($output)
+  }];
+}
+
+def ArrayRefFromArrayOp : ArcOp<"arrayref.from_array", [
+  MemoryEffects<[MemWrite]>,
+  AllTypesMatch<["input", "output"]>,
+  TypesMatchWith<"array type matches output type",
+                 "array", "output",
+                 "ArrayRefType::get("
+                   "cast<hw::ArrayType>($_self).getElementType(), "
+                   "cast<hw::ArrayType>($_self).getNumElements())">
+]> {
+  let summary = "arrayref from array";
+  let description = [{
+    Populates an arrayref with the elements of an array.
+  }];
+  let arguments = (ins ArrayRefType:$input, ArrayType:$array);
+  let results = (outs ArrayRefType:$output);
+  let assemblyFormat = [{
+    $input `=` $array attr-dict `:` type($output) `,` type($array)
+  }];
+}
+
+def ArrayRefToArrayOp : ArcOp<"arrayref.to_array", [
+  TypesMatchWith<"array type matches result type",
+                 "input", "result",
+                 "hw::ArrayType::get("
+                   "cast<ArrayRefType>($_self).getElementType(), "
+                   "cast<ArrayRefType>($_self).getNumElements())">
+]> {
+  let summary = "arrayref to array";
+  let description = [{
+    Creates an array from an arrayref.
+  }];
+  let arguments = (ins ArrayRefType:$input);
+  let results = (outs ArrayType:$result);
+  let assemblyFormat = [{
+    $input attr-dict `:` functional-type($input, $result)
+  }];
+}
+
 #endif // CIRCT_DIALECT_ARC_ARCOPS_TD

include/circt/Dialect/Arc/ArcPasses.td

@@ -177,6 +177,11 @@ def LowerClocksToFuncs : Pass<"arc-lower-clocks-to-funcs", "mlir::ModuleOp"> {
   let dependentDialects = ["mlir::func::FuncDialect", "mlir::scf::SCFDialect"];
 }
 
+def LowerArrays : Pass<"arc-lower-arrays", "mlir::ModuleOp"> {
+  let summary = "Lower arrays to use bufferized arrayrefs";
+  let dependentDialects = ["arc::ArcDialect"];
+}
+
 def LowerLUT : Pass<"arc-lower-lut", "arc::DefineOp"> {
   let summary = "Lowers arc.lut into a comb and hw only representation.";
   let dependentDialects = ["hw::HWDialect", "comb::CombDialect"];

include/circt/Dialect/Arc/ArcTypes.h

@@ -9,11 +9,25 @@
 #ifndef CIRCT_DIALECT_ARC_ARCTYPES_H
 #define CIRCT_DIALECT_ARC_ARCTYPES_H
 
+#include "circt/Dialect/HW/HWTypeInterfaces.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Types.h"
 
 #define GET_TYPEDEF_CLASSES
 #include "circt/Dialect/Arc/ArcTypes.h.inc"
 
+namespace circt {
+namespace arc {
+
+/// Compute the bit width a type will have when allocated as part of the
+/// simulator's storage. This includes any padding and alignment that may be
+/// necessary once the type has been mapped to LLVM. The idea is for this
+/// function to be conservative, such that we provide sufficient storage bytes
+/// for any type.
+std::optional<uint64_t> computeLLVMBitWidth(mlir::Type type);
+
+} // namespace arc
+} // namespace circt
+
 #endif // CIRCT_DIALECT_ARC_ARCTYPES_H

include/circt/Dialect/Arc/ArcTypes.td

@@ -11,9 +11,12 @@
 
 include "circt/Dialect/Arc/ArcDialect.td"
 include "mlir/IR/AttrTypeBase.td"
+include "mlir/IR/BuiltinTypeInterfaces.td"
 include "mlir/IR/OpBase.td"
+include "circt/Dialect/HW/HWTypeInterfaces.td"
 
-class ArcTypeDef<string name> : TypeDef<ArcDialect, name> { }
+class ArcTypeDef<string name, list<Trait> traits = []> :
+  TypeDef<ArcDialect, name, traits> { }
 
 def StateType : ArcTypeDef<"State"> {
   let mnemonic = "state";
@@ -56,4 +59,42 @@ def SimModelInstance : ArcTypeDef<"SimModelInstance"> {
   let assemblyFormat = "`<` $model `>`";
 }
 
+def ArrayRefType : ArcTypeDef<"ArrayRef", [
+  DeclareTypeInterfaceMethods<BitWidthTypeInterface>,
+  DeclareTypeInterfaceMethods<ShapedTypeInterface>
+]> {
+  let summary = "array reference";
+  let description = [{
+    An array reference is a reference (pointer) to an array of values.
+  }];
+  let mnemonic = "arrayref";
+  let parameters = (ins
+    "::mlir::Type":$elementType,
+    "uint64_t":$size
+  );
+
+  let builders = [
+    TypeBuilderWithInferredContext<(ins "Type":$elementType, "uint64_t":$size), [{
+      auto *ctx = elementType.getContext();
+      return $_get(ctx, elementType, size);
+    }]>
+  ];
+
+  let assemblyFormat = "`<` $size `` custom<XInDimList>($elementType) `>`";
+
+  let genVerifyDecl = 1;
+
+  let extraClassDeclaration = [{
+    size_t getNumElements() const;
+
+    std::optional<size_t> getByteWidth() const {
+      std::optional<size_t> bitWidth = getBitWidth();
+      if (!bitWidth.has_value()) {
+        return std::nullopt;
+      }
+      return (*bitWidth + 7) / 8;
+    }
+  }];
+}
+
 #endif // CIRCT_DIALECT_ARC_ARCTYPES_TD

include/circt/Tools/arcilator/pipelines.h

@@ -126,6 +126,10 @@ struct ArcToLLVMOptions : mlir::PassPipelineOptions<ArcToLLVMOptions> {
   Option<std::string> traceFileName{
       *this, "trace-file", llvm::cl::desc("Output file for signal traces"),
       llvm::cl::init("")};
+  Option<bool> bufferizeArrays{
+      *this, "bufferize-arrays",
+      llvm::cl::desc("Bufferize arrays before lowering to LLVM"),
+      llvm::cl::init(true)};
 };
 void populateArcToLLVMPipeline(mlir::OpPassManager &pm,
                                const ArcToLLVMOptions &options = {});