[FEATURE] Enable resource profiling for IM

[innobead]

          > How many bad backups is it?

168

Would it be interesting to take a look at profiling data to determine which part of the controller is eating CPU? https://longhorn.io/kb/troubleshooting-generate-pprof-runtime-profiling-data/

This will definitely be useful to me going forward. Thanks for the call out. However, it looks like most of the CPU is eaten up by instance-manager, which we don't appear to have enabled pprof for.

Originally posted by @ejweber in #6358 (comment)

[innobead]

Even though IM is system managed pod by Longhorn, but it still deserves profiling at runtime.

[shuo-wu]

One quick question, does this mean that we need a HTTP server inside IM pod for pprof?

[innobead]

It seems @Vicente-Cheng is working on this.

[Vicente-Cheng]

Maybe we could make it related to #6282
These would be similar.

[innobead]

@derekbit is working on it.

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

longhorn/longhorn-instance-manager#307

• Which areas/issues this PR might have potential impacts on?
  Area: profiling
  Issues

[roger-ryao]

Verified on master-head 20231229

• longhorn master-head 07bdb41
• longhorn-instance-manager master-head longhorn/longhorn-instance-manager@34c40f0

The test steps

1. Forward the port 6060 from the instance-manager pod to local port 6060:

   kubectl port-forward ${instance-manager-pod-name} -n longhorn-system 6060:6060
   

2. see the pprof debug web page: http://localhost:6060/debug/pprof/

Result Passed

1. pprof service works as expected.