Add VirtualInMemoryArray that keeps small arrays in memory

[tomwhite]

This is a continuation of the idea in #247 and #290 - there is no need to materialize arrays to Zarr if they can be computed from the block ID (#247 and #290) or if they are small enough to be in memory anyway (this PR).

I've also added a check that the arrays created in this way (via asarray) are smaller than 1MB. Anything bigger is best written to shared storage (typically as a Zarr file), rather than be serialized in the plan (which is currently what is happening).

[dcherian]

a check that the arrays created in this way (via asarray) are smaller than 1MB.

Perhaps you can avoid this with the "broadcast trick" (dask/dask#9517)

np.full(shape, 1) == np.ones(shape) == np.array([1]).broadcast_to(shape)

[tomwhite]

That's a clever trick, thanks for pointing it out @dcherian! In this case I don't think it applies, since we have a small array of arbitrary values, not a constant array created by full - but it might be something we can use in that case.

[tomwhite]

I just did a comparison of the number of Zarr arrays created during a unit test run:

• Main: 431 arrays
• This PR: 296 arrays

This probably isn't representative of real workloads since asarray isn't used much (and if it is, the arrays are small and don't dominate the computation).