Collector recovery from component panic

[djaglowski]

Is your feature request related to a problem? Please describe.
There have been a number of issues in the past where the collector crashes due to an unexpected panic in a component. A couple examples:

• [receiver/mongodbatlas] Panic in collecting event logs opentelemetry-collector-contrib#19191
• datadog exporter panic opentelemetry-collector-contrib#18099

How can the collector be more resilient to panics that originate within components?

• Are there any panics that we can fully recover from?
• Assuming there are at least some panics that are properly fatal, is there a strategy that we should use to crash more elegantly?

Describe the solution you'd like
Recovery from a panic is only possible within the goroutine where the panic originated. Therefore, I think we can look at two separate cases:

1. Component Start and Shutdown funcs are called synchronously within the service package. Potentially we could recover in a useful way here.
2. Components often start their own goroutines. If there is a recovery behavior we would like to encourage, I think the best we can do is to export a recovery function from a common package and encourage that each new goroutine should call this function immediately before doing anything else. This might be overkill in some cases, but perhaps there is a balance. e.g. call this func for long running or complex goroutines.

Describe alternatives you've considered
open-telemetry/opentelemetry-collector-contrib#16598 is a possible strategy for crashing more elegantly.

[mx-psi]

Relates to #3955 (or duplicates it?)

[djaglowski]

Thanks for pointing out the duplicate @mx-psi. I'll close this one and update the tracking issue.