optimising usage of StructureTraceGraph

[kotharironak]

We have observed performance issues due to the re-construction need for StructureTraceGraph. Some more details found in a workaround fix in the PR - #100

As part of this issue, we would like to see if can we only use a single instance of StructureTraceGraph for a given trace?

[findingrish]

Currently we build a StructuredTraceGraph from StructuredTrace in the following enrichers:

• ApiBoundaryTypeAttributeEnricher
• DefaultServiceEntityEnricher
• BackendEntityEnricher

Need for rebuilding of the graph should be evaluated.
There seem to be 2 reasons that we rebuild this graph:

• We don't pass around state between enrichers
• StructuredTrace itself changes in the enrichers

My proposal is to create a graph right at the beginning and pass around this Graph to the other enrichers.
Any enrichment to the StructuredTrace needs to be reflected on the already built StructuredTraceGraph, by updating it accordingly. The feasibility of this needs to be checked.

Lastly we should checkin the problematic trace in the repo itself (if not the exact trace then a derivative) and add a micro-benchmarking to control and improve the latency of the enrichers.
cc @buchi-busireddy @skjindal93 @jcchavezs

[GurtejSohi]

Observed an issue which is related to this. So, mentioning it here:

Currently, at some places in StructuredTraceGraph, we're using the whole event (or entity) as the key in hash maps or hash sets, which means every single access(put and get) to these maps involves computing a hash code of the whole event which can be quite expensive. Instead, we can use the event id (or the entity id) as the key which would significantly bring down the cost of accessing an element in the map and thus, bring down the cost of building a StructuredTraceGraph as well.

I've raised a PR for fixing this issue: hypertrace/data-model#14 .

[kotharironak]

Closing this, as this was done by introducing state as part of ThreadLocal to reuse StructureTraceGraph.