Mutex contention in `servicegraphs` processor

[flxbk]

In some circumstances, the servicegraph processor in the metrics generator can spend a lot of time locking and unlocking the mutex in UpsertEdge.

The following is a screenshot from two execution traces captured from metrics generators ingesting from Kafka. Both show one of the goroutines running readCh. The bottom trace shows a state where the generator is keeping up with its backlog, while at the top it's falling behind.

Execution trace goroutine flamegraph

If we need to improve throughput, optimizing this locking could be an option.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.