chore: improve tracing in network monitor

[SantiagoPittella]

This PR uses main as base branch

Improves the instrumentation in the miden-network-monitor binary.

This is the current view of the traces (in jaeger):

[SantiagoPittella]

@bobbinth @Mirko-von-Leipzig mdbook publishing a new, non compatible, version a few hours ago. I tried removing the multilingual option but other errors arised. Should I lock the Ci version to the previous (v0.4.52) until we complete the migration? mdbook people provides a migration guide https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#05-migration-guide

[igamigo]

Not sure if the plan is still to merge this and patch release it, but LGTM! Sorry for the delay in the review.
Not super familiar with the frequencies of some processes. Doesn't seem like these changes would be very spammy but it's probably something to watch out for.

[igamigo]

nit: Emitting 2 events here feels a bit redundant/unnecessary

[SantiagoPittella]

I removed the first one. For the debug! level I left one log per component.

[igamigo]

Something that could be nice for troubleshooting is to raise info events for service status changes. Something like

  let mut last = Status::Unknown;
  // ...
  let status = check_remote_prover_status(...).await;
  if status.status != last {
      info!(target: COMPONENT, prover = %name, status = ?status.status, "Remote prover status changed");
      last = status.status;
  }

But maybe this is already easily filterable with the current traces

[SantiagoPittella]

In line 443 we have this:

   debug!(target: COMPONENT, prover_name = %name, remote_prover_status = ?status, "Remote prover status check successful");

Is that what you meant?

[igamigo]

Mostly meant emitting events when the status changes (so for example, when a service goes from healthy to unhealthy or viceversa). It feels like it could be useful for troubleshooting, but maybe not.

[SantiagoPittella]

Probably, but the idea of the monitor is not troubleshooting but a quick glance of the overall state of the components

[SantiagoPittella]

Not sure if the plan is still to merge this and patch release it, but LGTM! Sorry for the delay in the review.

Yeah, maybe now we want to change the base to next and avoid this release + deploy cc @bobbinth

Not super familiar with the frequencies of some processes. Doesn't seem like these changes would be very spammy but it's probably something to watch out for.

Maybe this can be controlled changing the sample rate if we decide that we need to reduce it? That will require some probably minor changes in the utils crate.