inference: refactor the core loops to use less memory

[aviatesk]

A minimum version of #43999, without any apparent regressions.

The original PR description by @Keno will follow:

---

Currently inference uses O(<number of statements>*<number of slots>) state
in the core inference loop. This is usually fine, because users don't tend
to write functions that are particularly long. However, MTK does generate
functions that are excessively long and we've observed MTK models that spend
99% of their inference time just allocating and copying this state.
It is possible to get away with significantly smaller state, and this PR is
a first step in that direction, reducing the state to O(<number of basic blocks>*<number of slots>).
Further improvements are possible by making use of slot liveness information
and only storing those slots that are live across a particular basic block.

The core change here is to keep a full set of slottypes only at
basic block boundaries rather than at each statement. For statements
in between, the full variable state can be fully recovered by
linearly scanning throughout the basic block, taking note of
slot assignments (together with the SSA type) and NewVarNodes.

The current status of this branch is that the changes appear correct
(no known functional regressions) and significantly improve the MTK
test cases in question (no exact benchmarks here for now, since
the branch still needs a number of fixes before final numbers make
sense), but somewhat regress optimizer quality (which is expected
and just a missing TODO) and bootstrap time (which is not expected
and something I need to dig into).

---

@nanosoldier runbenchmarks(!"scalar", vs=":master")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here.

[aviatesk]

@nanosoldier runtests(ALL, vs = ":master")

[aviatesk]

@nanosoldier runbenchmarks("broadcast" || "collection", vs=":master")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here.

[nanosoldier]

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.

[aviatesk]

@nanosoldier runtests(["LoopVectorization", "RecursiveFactorization"], vs = ":master")
@nanosoldier runbenchmarks(!"scalar", vs=":master")

[nanosoldier]

Your package evaluation job has completed - no new issues were detected. A full report can be found here.

[ianatol]

LGTM! Good work Shuhei!

Edit: Just saw PkgEval results, I guess there is more work to be done. Let me know if you want help, as I read it somewhat thoroughly to do this review (though obviously not thoroughly enough to catch any bugs 😂 )

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here.

[aviatesk]

@nanosoldier runtests(["LoopVectorization", "RecursiveFactorization"], vs = ":master")

[nanosoldier]

Your package evaluation job has completed - no new issues were detected. A full report can be found here.

[aviatesk]

@nanosoldier runtests(ALL, vs = ":master")
@nanosoldier runbenchmarks("inference", vs=":master")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here.

[nanosoldier]

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.

[aviatesk]

@nanosoldier runtests(ALL, vs = ":master")
@nanosoldier runbenchmarks(!"scalar", vs=":master")

[aviatesk]

@nanosoldier runbenchmarks("inference", vs=":master")

[nanosoldier]

Something went wrong when running your job:

NanosoldierError: failed to run benchmarks against primary commit: failed process: Process(`sudo -n /nanosoldier/cset/bin/cset shield -e -- sudo -n -u nanosoldier-worker -- /nanosoldier/workdir/jl_P8M1Ku/benchscript.sh`, ProcessExited(1)) [1]

Logs and partial data can be found here

[aviatesk]

@nanosoldier runbenchmarks("inference", vs=":master")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here.

[aviatesk]

@nanosoldier runbenchmarks("inference", vs=":master")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here.

[aviatesk]

The nanosolidier results look nice. Going to merge.