[MLIR] Improve concurrency when using async.

[erick-xanadu]

Context: The initial implementation of async had a naive algorithm that serialized a lot of potentially concurrent code. This PR improves the algorithm and guarantees maximum concurrency.

Description of the Change:

The async.execute instruction returns a token and a promise.

 %token, %promise = async.execute {
   // body...
   async.yield %0 : !some.type
}

We can then wait on the promise or the token

async.await %token // doesn't return a value but waits for all execution in region to finish
%val = async.await %promise // does return a value, whatever is yielded

If async.await is used for the first time and the value is not available, the thread will wait
until the value becomes available. If the value is available, the function returns practically immediately. For example.

%1 = async.await %promise // promise is not yet available, will wait.
%2 = async.await %promise // promise is now available, will not wait.

This makes the job of deciding where to place awaits really easy.
We just need to place awaits just before every use of the original value.
For example, take the following piece of code

%quantum_results = call @qnode

// Arbitrary control flow
// This use might be inside a different basic block.
%some_val = some.op %quantum_results

First we translate the call to an async.execute (which was performed before this
function was called) When this function was called, the transformation is not yet
complete. Because there are still uses of %quantum_results.

%async_quantum_results = call @qnode

// Arbitrary control flow
// This use might be inside a different basic block.
%some_val = some.op %quantum_results

This function will perform the following change:

%async_quantum_results = call @qnode

// Arbitrary control flow
// This use might be inside a different basic block.
%await_quantum_results = async.await %async_quantum_results
%some_val = some.op %await_quantum_results

And it will do so for every use of %quantum_results. Eg.

%async_quantum_results = call @qnode

// Arbitrary control flow
// This use might be inside a different basic block.
%await_quantum_results_0 = async.await %async_quantum_results
%some_val_0 = some.op %await_quantum_results_0
%await_quantum_results_1 = async.await %async_quantum_results
%some_val_1 = some.op %await_quantum_results_1

Benefits: Improved concurrency.

Possible Drawbacks: Not really a drawback, but potential improvement point (which is independent of this commit). We could have a heuristic to decide when async is profitable locally. I believe async is always profitable in remote execution, but not locally.

[sc-52637]

[rmoyard]

Thanks, great solution!

[maliasadi]

Thanks @erick-xanadu! I have a few questions..