Add documentation for hot/cold splitting (dotnet#73029)

* Add documentation for hot/cold splitting

* Address feedback

* Improve phrasing

Author: Aman Khalid

docs/design/coreclr/jit/hot-cold-splitting.md

@@ -0,0 +1,173 @@
+# Hot/Cold Code Splitting
+
+This document describes the current state of hot/cold splitting in the JIT.
+
+Hot/Cold splitting is an optimization that splits code into frequently-executed ("hot") and rarely-executed ("cold")
+parts, and places them in separate memory regions. Increased hot code density better leverages spatial locality,
+improving application performance via fewer instruction cache misses, less OS paging, and fewer TLB misses.
+
+## Background
+
+The JIT previously supported hot/cold splitting for AOT-compiled NGEN images in .NET Framework. With Crossgen2 support
+[in progress](https://github.com/dotnet/runtimelab/tree/feature/hot-cold-splitting) (and no existing support for
+splitting dynamically-generated code), JIT support has not been tested since retiring .NET Framework -- thus, there
+are likely regressions. Furthermore, the JIT never supported splitting functions with certain features, like exception
+handling or switch tables. Finally, with ARM64 code generation being a newer addition to the JIT, hot/cold splitting
+was never implemented for the architecture. These limitations significantly inhibit the applicability of hot/cold splitting.
+
+The below sections describe various improvements made to the JIT's hot/cold splitting support to remove such limitations.
+
+## Testing the JIT Without Runtime Support
+
+Without runtime support for hot/cold splitting in .NET as of summer 2022, testing the JIT's existing hot/cold splitting
+support is not as simple as turning the feature on. A new "fake" splitting mode, enabled by the
+`COMPlus_JitFakeProcedureSplitting` environment variable, removes this dependency on runtime support. This mode allows
+the JIT to execute its hot/cold splitting workflow without changing the runtime's behavior. This workflow proceeds as
+follows:
+
+* The JIT identifies where to split the function in `Compiler::fgDetermineFirstColdBlock`, as usual.
+* In `Compiler::eeAllocMem`, the JIT requests one memory buffer from the host (either Crossgen2 or the VM) for the
+entire function; this is unlike normal splitting, where separate buffers are allocating for the hot/cold sections.
+  * After the host has allocating the buffer, the JIT manually sets the cold code pointers to right after the hot section.
+  * Note there is no space between the hot/cold sections, unlike with normal splitting. The instructions are still contiguous.
+* During code generation, the JIT emits instructions as if the hot/cold sections are arbitrarily far away:
+  * Jumps between hot/cold sections are long.
+  * The JIT reports jump target relocations to the host as necessary.
+  * On some platforms like ARM64 (see below), the JIT emits certain pseudo-instructions to handle the instruction section's
+lack of contiguousness.
+* For the sake of simplicity, the JIT generates unwind info as if it is not splitting. Because the hot/cold sections
+are adjacent, the JIT generates unwind info once for the entire function.
+
+While enabling fake-splitting also enables `opts.compProcedureSplitting`, there is no guarantee the JIT will fake-split
+a function unless `Compiler::fgDetermineFirstColdBlock` finds a splitting point; without PGO data, the JIT's heuristics
+may be too conservative for extensive testing. To aid regression testing, the JIT also has a stress-splitting mode now,
+under `COMPlus_JitStressProcedureSplitting`. When `opts.compProcedureSplitting` and stress-splitting are both enabled,
+the JIT splits every function after its first basic block; in other words, `fgFirstColdBlock` is always
+`fgFirstBB->bbNext`. The rest of the hot/cold splitting workflow is the same: The JIT emits instructions to handle the
+split code sections and, if fake-splitting, utilizes only one memory buffer.
+
+When used in tandem, fake-splitting and stress-splitting have strong potential to reveal regressions in the JIT's
+hot/cold splitting functionality without runtime support. As such, a new rolling test job in the
+[runtime-jit-experimental](https://dev.azure.com/dnceng/public/_build?definitionId=793), `jit_stress_splitting`, runs
+all `dotnet/runtime` tests with fake-splitting and stress-splitting enabled.
+
+### PRs
+
+* [runtime/69763](https://github.com/dotnet/runtime/pull/69763): Implement fake-splitting and stress-splitting modes
+* [runtime/69922](https://github.com/dotnet/runtime/pull/69922): Add `jit_stress_splitting` to `runtime_jit_experimental`
+
+## ARM64 Support
+
+After devising strategies for testing the JIT independently of runtime support for splitting, achieving functional
+parity for ARM64 became a priority. While initial splitting prototypes in Crossgen2 target x64, the JIT can
+achieve some correctness with hot/cold splitting on ARM64 by leveraging fake-splitting alone.
+
+Most of the JIT's hot/cold splitting workflow is architecture-independent; only code generation is ARM64-specific.
+The majority of implementation work here is thus related to emitting various long pseudo-instructions:
+
+* On both ARM32 and ARM64, conditional jumps have less range than unconditional jumps due to the architectures' fixed
+instruction width. Normally, the conditional jump's range is large enough to cover any reasonably-sized function.
+With splitting enabled, hot/cold sections can be arbitrarily far apart for dynamically-generated code, and up to
+2<sup>32</sup> bits apart in AOT-compiled code (this is the maximum code size allowed in PE files). To avoid
+arbitrarily limiting code sizes, conditional jumps must have the same range as unconditional jumps. "Jump stubs" solve
+this by replacing each conditional jump with a negated conditional jump, followed by an unconditional jump to the
+original target -- this pseudo-instruction's format is `IF_LARGEJMP`. For example, `branch condition, target` becomes
+the following:
+
+```
+branch !condition, pc+1
+branch target
+```
+
+* Without splitting, the read-only data section is adjacent to the function's instruction section on ARM64. When
+splitting, the data section is adjacent to the hot section; from the hot section, we can load constants with a single
+`ldr` instruction. However, this is not possible from the cold section: Because it is arbitrarily far away, the target
+address cannot be determined relative to the PC. Instead, the JIT emits a `IF_LARGELDC` pseudoinstruction with a
+few different possibilities:
+  * First, compute the target page address with an `adrp` instruction.
+  * Case 1: Load the constant into a general register with a `ldr` instruction. (Final sequence: `adrp + ldr`)
+    * If the destination register is a vector register, move the value from the general register with a `fmov`
+instruction. (Final sequence: `adrp + ldr + fmov`)
+  * Case 2: If the constant is 16 bytes in size, load it directly into a vector register.
+    * General registers are 8 bytes in width on ARM64. Thus, they cannot temporarily hold the constant.
+    * Compute the exact address with an `add` instruction, and load the constant with an `ld1` instruction.
+(Final sequence: `adrp + add + ld1`)
+
+Aside from these pseudo-instructions, hot/cold splitting required a few other tweaks to ARM64 code generation:
+* When emitting long jumps between hot/cold sections, the JIT reports the target's relocation to the host with the
+relocation type `IMAGE_REL_ARM64_BRANCH26`.
+* While enabling fake-splitting did not require changes here, it is worth noting an importance difference in unwind info
+generation on x64 versus ARM64. On x64, the JIT emits the full unwind info for a hot function fragment, and emits
+"chained" unwind info for the cold function fragment. This chained unwind info does not contain any unwind codes, but
+instead points to the hot fragment's unwind info. When unwinding, the VM will use this chained info to find the relevant
+unwind info.
+
+There is no concept of chained unwind info on ARM64; instead, the JIT generates unwind info for each function fragment,
+regardless of its hot/cold status. While this should not have any immediate implications for JIT work around hot/cold
+splitting, this does affect the feature's implementation in Crossgen2 and the VM. On x64, the Crossgen2 splitting
+prototype uses chained unwind info to differentiate between cold main body fragments and cold EH funclets (see below for
+details on EH splitting). This comparison is not possible on ARM64 -- the JIT may have to pass more information to the
+host when generating unwind info on ARM64 to indicate if a cold fragment is a funclet.
+
+### PRs
+
+* [runtime/70708](https://github.com/dotnet/runtime/pull/70708): Enable fake-splitting on ARM64
+
+## Splitting Functions with Exception Handling (EH)
+
+An EH funclet is a "mini-function" for handling or filtering exceptions; for example, for a conventional "try/catch"
+expression, the catch block becomes a funclet (this is true for finally/fault/filter/etc. blocks as well). The
+JIT places EH funclets contiguously in memory, adjacent to the main function body. Because of the prevalence of
+exception handling in .NET programs, enabling splitting of EH funclets massively expands this optimization's
+applicability.
+
+Because EH funclets immediately succeed the main function body, the JIT can easily split such functions without
+breaking existing invariants:
+
+* If the JIT finds a split point in the main body, it splits there as usual. The latter part of the main body,
+along with all of the function's EH funclets, is cold.
+* If the JIT does not find a split point in the main body, and none of the funclets execute frequently, it splits
+at the beginning of the funclet section. The main body is hot, and all EH funclets become cold.
+* Else, no splitting occurs.
+
+This approach may not be the most performant implementation: Splitting funclets individually could yield better
+spatial locality. However, this would require re-arranging the order of funclets (currently, there is no specific
+order imposed), and significantly altering unwind info generation, thus breaking many invariants in the host. This
+approach enables splitting in many more scenarios without breaking existing invariants or introducing
+architecture-specific workarounds. However, if the JIT supports splitting functions multiple times in the future, we
+should revisit this.
+
+In the absence of PGO data, the JIT assumes exceptions occur rarely; this justifies moving handlers to the cold section.
+Because `finally` blocks execute regardless of an exception occurring, it may be detrimental to make these handlers
+cold. Thus, [Compiler::fgCloneFinally](https://github.com/dotnet/runtime/blob/41419131095d36fb5b811600ad0dab3b0d804269/src/coreclr/jit/fgehopt.cpp#L617)
+copies the `finally` block to the hot section, provided it is not too large. Once runtime support for splitting matures,
+we should revisit this optimization to ensure the JIT is not too sparse or overzealous in its usage.
+
+### PRs
+
+* [runtime/71236](https://github.com/dotnet/runtime/pull/71236): Enable hot/cold splitting of EH funclets
+* [runtime/71273](https://github.com/dotnet/runtime/pull/71273): Disable `HANDLER_ENTRY_MUST_BE_IN_HOT_SECTION`
+* [runtimelab/1923](https://github.com/dotnet/runtimelab/pull/1923): Fix unwind info for cold EH funclets on x64
+* [runtimelab/1930](https://github.com/dotnet/runtimelab/pull/1930): Fix unwind info for cold EH funclets on ARM64
+
+## Future Work
+
+As of writing, support for hot/cold splitting in Crossgen2 on x64 is in progress. While some future tasks are JIT-specific
+and will not require runtime support to begin work, many will require close collaboration. See the `dotnet/runtimelab`
+hot/cold splitting [prototype](https://github.com/dotnet/runtimelab/tree/feature/hot-cold-splitting) for
+runtime-specific tasks.
+
+* Profile runtime effects of hot/cold splitting. Since we are largely interested in how this affects spatial locality,
+key metrics could include number of hot/cold page touches, number of jumps to the cold section taken, number of
+instruction cache misses, etc. It is important that such profiling utilizes PGO data, as the JIT's splitting heuristics
+are quite sparse, and may not be useful for measuring performance.
+* Enable hot/cold splitting of functions with switch tables.
+* Work with Crossgen2 prototype to support hot/cold splitting on ARM64.
+  * This task will specifically require work in the JIT for differentiating cold funclets from regular cold code.
+On x64, Crossgen2 and the VM use chained unwind info (or lack thereof) to differentiate the two. Since there is no
+concept of chained unwind info on ARM64, the JIT may need to report more information to the host.
+* Support hot/cold splitting of dynamically-compiled code.
+  * Since the JIT has historically never supported splitting jitted code, it may be interesting to measure the overhead
+of performing hot/cold splitting during runtime.
+* Support hot/cold splitting of NativeAOT code.
+  * Most of the work here will likely involve generating unwind info correctly.