[EEM] Discrepancy between throughput metric from service entity definition and apm-server

[kpatticha]

Issue

kibana/x-pack/plugins/observability_solution/entity_manager/server/lib/entities/built_in/services.ts

Lines 74 to 83 in 8c5b085

	name: 'throughput',
	equation: 'A',
	metrics: [
	{
	name: 'A',
	aggregation: 'doc_count',
	filter: serviceTransactionFilter(),
	},
	],
	},

throughput from data transforms 360tpm while from apm-server is only 3tpm for the same service

Screen.Recording.2024-07-19.at.18.52.14.mov

[elasticmachine]

Pinging @elastic/obs-knowledge-team (Team:obs-knowledge)

[klacabane]

From my initial testing against oblt clusters I could find discrepancies but not of that order. The testing was mostly focused on history dataset so I wonder if there could be something fishy happening in the latest dataset, I'll have a look.
@kpatticha if you can reproduce this easily, could you check if the throughput in the related history documents is also much higher ? Do the service.name: only-apm-0 history documents also have ~300tpm ?

[klacabane]

I think the metric is just wrong, we should instead value_count on transaction.duration.summary