Service Remoting emitted performance counters are not easy to collect using WAD2AI

[nizarq]

Service Remoting emits performanc counters like below:

Category name	Counter name	Description
Service Fabric Service	# of outstanding requests	Number of requests being processed in the service
Service Fabric Service	Average milliseconds per request	Time taken (in milliseconds) by the service to process a request
Service Fabric Service	Average milliseconds for request deserialization	Time taken (in milliseconds) to deserialize service request message when it is received at the service
Service Fabric Service	Average milliseconds for response serialization	Time taken (in milliseconds) to serialize the service response message at the service before the response is sent to the client

The best way to collect these counters using WAD is using wildcards e.g.,

"counterSpecifier": "\\Service Fabric Service(*)\\Average milliseconds per request",

However, counters containing wildcards are not picked up by WAD2AI. As it is unrealistic to expect customers to know in advance all the services that might run on a service fabric node (as opposed to PaaS V1), it is unrealistic to ask them to mitigate the situation by explicitly spelling out all possible values at configuration time. WAD2AI should start supporting performance counters with wildcards.

[johnkattenhorn]

Hey @nizarq - we found that we could pick these up by deploy OMS Agent ... however we can see the performance counters but the CounterValue is always zero.

[nizarq]

Hi @johnkattenhorn - I don't really have any OMS agent experience.
May be @keikhara can answer how performance counter collection happens using the OMS agent and how you can troubleshoot.