Adding an executable environment and plumbing through processor information.

[benvanik]

This adds an iree_hal_executable_environment_*_t struct to wrap up the existing import table, a new processor info struct, and a new reserved spot for specialization constants.

An "executable environment" is now provided to executables as they are loaded as well as on each dispatch invocation. On initial query the executable code can use the optional processor information, import functions, and executable-level specialization constants to decide which implementation of functions to use by switching the pointers returned in the library table. There may be future work around that area to support different concurrently specialized versions of the same library but for today our usage doesn't trigger that issue. Dispatch functions receive the same processor information, imports, and constants and can be used to enable per-invocation specialization (switch paths based on the current processor microarchitecture when in a heterogenous system, use a runtime-derived executable-level constant computed by the host code such as workgroup sizes, etc).

The general equivalence to regular software is that flags provided to clang become HAL configurations in iree-translate, runtime queries done once to switch between implementations compiled in become specialization constants, and queries done per invocation become the environment/processor info. This design gives us the same capabilities normal software has while preserving our intra-architecture artifact portability and forward compatibility along with the ability to separate host and device (running in enclaves, on remote devices, etc).

Isolation is being maintained between the internal parts of the runtime that only need be source compatible (such as how to query the current processor ordinal via a platform/arch-specific impl) and those that bridge into the compiler. The executable library structs are part of the public stable ABI for the executables and once we go stable will only be able to be appended or versioned coarsely. By construction it's not possible to load an executable compiled for any other major architecture than the one doing the loading so each can have its own data format and there's no need to normalize all the various architectural quirks - instead the runtime is just passing data from some function that produces the data to the executable using it by memcpy. Beyond keeping things simple this also allows the runtime to be architecture-agnostic with respect to out-of-tree ISA's/OS'/bare-metal HAL's/etc; there's no enum of architectures or list of features in the runtime that needs to change beyond custom data query functions that can be overlaid into the build.

The processor data itself is represented as a set of opaque uint64_t fields. One could just store the architectural info registers right in the data, though of all the thousands of bits there are only a handful we care about and we can append to the set as we need it. A benefit of the arch registers would be that the runtime does not need to be updated to run models using newer features, but that's risky (it may mean the kernel or hosting code doesn't support it - like trying to use avx512 in an app compiled for avx2). The arch registers may not always be available to query either (sandboxing/fingerprinting avoidance), and having something that was robust to partial information is important. Another example use of the fields is a bitmap indicating which logical processors are running on a little core in a big.LITTLE setup such that testing per-invocation which code path should be used becomes (processor.data[CORE_TYPE_FIELD] >> processor_id) & 1. All other information about cache topology and such can come through specialization constants queried from the host side as that's what needs to adjust workgroup parameters anyway. The first few bits we add will likely be those from iree/tools/utils/cpu_features.h.

Note that the intent here is that executables we produce from the compiler always run on some baseline; unless a user is hyper optimizing for size the additional fallback paths would at most double the small executable size but realistically only increase it a marginal amount: not every dispatch is going to use all of the conditional features and need to be duplicated. There's going to need to be some compiler infra to make that happen, though (plumbing in MLIR through to LLVM's function multi-versioning), so today we're probably stuck with whatever the user specifies to the compiler :(

Progress on #5417. Follow-on PRs will fold in the iree/tools/utils/cpu_features.h work though there will need to be some discussion about how to represent it. Prepares for #8469.
Work remains to wire up the ability to specify the specialization constants in the compiler/runtime HAL layers; what's here is just the plumbing for CPU.

[iree-github-actions-bot]

Abbreviated Benchmark Summary

@ commit 2412f882f2f096beaef70726bd6f9e18ed9768f4 (vs. base c3cefa5151e0d94d08dbf2ed9019594df24dd3fd)

Improved Benchmarks 🎉

Benchmark Name	Average Latency (ms)	Median Latency (ms)	Latency Standard Deviation (ms)
MobileNetV3Small [fp32,imagenet] (TFLite) big-core,full-inference,experimental-flags with IREE-Dylib-Sync @ Pixel-4 (CPU-ARMv8.2-A)	14 (vs. 15, 6.67%↓)	14	1
PoseNet [fp32] (TFLite) full-inference,experimental-flags with IREE-Vulkan @ Pixel-6-Pro (GPU-Mali-G78)	16 (vs. 17, 5.88%↓)	16	0

For more information:

• Full benchmark result tables
• Buildkite build

[benvanik]

(this intentionally doesn't decide what information we give to executables or how we use it in codegen, but defines how we move that information from the runtime into executable code)

[MaheshRavishankar]

THanks Ben. I cant really grock all the details here. I know the intent here. I'll play around with this a bit more and sync with you if I have any questions.

[benvanik]

No problem - you can consider it mostly FYI that this is now possible to use. It's not taking any opinions about the actual data used or how it integrates into codegen as I figured that'd be a bigger design discussion - at the end of the day all this does is memcpy a blob :P