Move the metrics recorder to a blocking thread to avoid hangs

[teor2345]

Scheduling

We should only do this ticket if we have fixed the other Zebra hang bugs, and Zebra still hangs frequently.

This is unlikely to be causing our hang issues, so we should do it last.

Is your feature request related to a problem? Please describe.

Zebra runs the metrics recorder on an async thread, which can block:

• async code running concurrently in the same task, and
• async code from other tasks on the same thread.

Describe the solution you'd like

Instead, Zebra should:

• use tokio's spawn_blocking function to run this task, and
• make all callers into async functions (or access the executor directly)

https://docs.rs/tokio/1.6.0/tokio/task/fn.spawn_blocking.html

Describe alternatives you've considered

We can use tokio's block_in_place function. This function only blocks concurrent code in the same task, which is better, but still not great.

We could also use block_in_place as we transition code.

Additional context

Blocking functions might be the source of hangs or slowdowns in the inbound service, or state service, or unrelated tasks.

[teor2345]

@mpguerra this change is high-risk, we should only do it after NU5 activation, and after upgrading to tokio 1.0

[teor2345]

This doesn't seem to cause problems in practice.