Metrics support for EdgeHub #1290

varunpuranik · 2019-06-03T19:14:29Z

Remove existing metrics from EdgeHub codebase
Remove existing metrics infrastructure in Util
Add new generic metrics infrastructure in Util
Add support for prometheus style metrics
Add messaging metrics to EdgeHub
Other metrics coming up in next PR.

myagley · 2019-06-06T16:01:36Z

Is it possible to add a dump from the http endpoint to see what the metrics are reported as to this PR description?

myagley · 2019-06-06T15:58:11Z

edge-util/src/Microsoft.Azure.Devices.Edge.Util/Microsoft.Azure.Devices.Edge.Util.csproj

    <PackageReference Include="App.Metrics.Reporting.InfluxDB" Version="3.0.0" />
    <PackageReference Include="App.Metrics.Reporting.TextFile" Version="3.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging" Version="2.2.0" />
    <PackageReference Include="Newtonsoft.Json" Version="12.0.1" />
    <PackageReference Include="Nito.AsyncEx" Version="5.0.0-pre-05" />
+    <PackageReference Include="OpenCensus" Version="0.1.0-alpha-42253" />
+    <PackageReference Include="OpenCensus.Exporter.Prometheus" Version="0.1.0-alpha-42253" />


Are these the latest versions? I understand open census is still early, but is there a non dev version of app.metrics?

I forgot to remove these.. we don't need them..

myagley · 2019-06-06T16:12:35Z

edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs

+        static class Metrics
+        {
+            static readonly IMetricsTimer MessagesTimer = Util.Metrics.Metrics.Instance.CreateTimer(
+                "message_send_latency_milliseconds",


Per the best practice, we should probably standardize on seconds?

message_send_duration_seconds

myagley · 2019-06-06T16:13:48Z

edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs

@@ -234,6 +235,86 @@ async Task HandleTwinOperationException(string correlationId, Exception e)
            }
        }

+        #region IDeviceProxy
+
+        public Task SendC2DMessageAsync(IMessage message) => this.underlyingProxy.SendC2DMessageAsync(message);


I'm trying to figure out why this diff is so big and what changed. Is this new code?

No, I just moved product code to be above the Events class (it got moved around during the cleanup sometiem back). No changes here other than Metrics class.

myagley · 2019-06-06T16:16:19Z

edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/RoutingEdgeHub.cs

-
-            public static IDisposable MessageLatency(IIdentity identity) => Util.Metrics.Latency(GetTags(identity), EdgeHubMessageLatencyOptions);
+            static readonly IMetricsHistogram MessagesHistogram = Util.Metrics.Metrics.Instance.CreateHistogram(
+                "message_size",


message_size_bytes?

Yes, changed.

myagley · 2019-06-06T16:17:11Z

edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs

-            }
-        }
+            static readonly IMetricsHistogram MessagesProcessLatency = Util.Metrics.Metrics.Instance.CreateHistogram(
+                "message_process_latency_milliseconds",


message_process_duration_seconds

myagley · 2019-06-06T16:17:31Z

edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs

-                Task completedTask = await Task.WhenAny(taskCompletionSource.Task, Task.Delay(MessageResponseTimeout));
-                if (completedTask != taskCompletionSource.Task)
+            static readonly IMetricsTimer MessagesTimer = Util.Metrics.Metrics.Instance.CreateTimer(
+                "message_send_latency_milliseconds",


message_send_duration_seconds

myagley · 2019-06-06T16:18:56Z

edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/linkhandlers/ModuleMessageLinkHandler.cs

+        static class Metrics
+        {
+            static readonly IMetricsMeter SentMessagesMeter = Util.Metrics.Metrics.Instance.CreateMeter(
+                "messages_sent",


messages_sent_total?

So, if we use a meter, AppMetrics adds _total to the name! :)

myagley · 2019-06-06T16:19:19Z

edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/linkhandlers/EventsLinkHandler.cs

+        static class Metrics
+        {
+            static readonly IMetricsMeter MessagesMeter = Util.Metrics.Metrics.Instance.CreateMeter(
+                "messages_received",


messages_received_total?

myagley · 2019-06-06T16:21:53Z

edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/appsettings_hub.json

+  "metrics": {
+    "enabled": true,
+    "listener": {
+      "port": 80,


We should probably try to allocate a port here: https://github.com/prometheus/prometheus/wiki/Default-port-allocations and then default to it. For now, we can start with something around 9600

cool, sounds good

myagley · 2019-06-06T16:22:39Z

edge-util/src/Microsoft.Azure.Devices.Edge.Util/Metrics.cs

@@ -12,11 +12,11 @@ namespace Microsoft.Azure.Devices.Edge.Util
    using App.Metrics.Timer;
    using Microsoft.Extensions.Configuration;

-    public static class Metrics
+    public static class Metrics2


Is this the old one?

myagley · 2019-06-06T16:23:09Z

edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/IEdgeMetrics.cs

+{
+    public interface IEdgeMetrics
+    {
+        void InitPrometheusMetrics(int port);


InitMetrics? We should probably find a way to remove Prometheus

Emm, reason I kept Prometheus in the name was because it also starts a http server.. The idea was to have different methods for push/pull modes..

Maybe I should rename it to InitMetricsListener..

varunpuranik · 2019-06-06T18:58:38Z

Here is the response from the Http endpoint..

# HELP edgehub_gettwin_total 
# TYPE edgehub_gettwin_total counter
edgehub_gettwin_total{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101"} 1

# HELP edgehub_messages_received_total 
# TYPE edgehub_messages_received_total counter
edgehub_messages_received_total{Protocol="Amqp",id="d101/Sender0",edgeDevice="d101"} 82

# HELP edgehub_messages_sent_total 
# TYPE edgehub_messages_sent_total counter
edgehub_messages_sent_total{Protocol="Amqp",from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101"} 81

edgehub_messages_sent_total{to="IoTHub",from="d101/Sender0",edgeDevice="d101"} 81

# HELP edgehub_reported_properties_update_total 
# TYPE edgehub_reported_properties_update_total counter
edgehub_reported_properties_update_total{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101"} 5

# HELP edgehub_message_process_duration_milliseconds 
# TYPE edgehub_message_process_duration_milliseconds summary
edgehub_message_process_duration_milliseconds_sum{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101"} 7112
edgehub_message_process_duration_milliseconds_count{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101"} 81
edgehub_message_process_duration_milliseconds{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101",quantile="0.5"} 134
edgehub_message_process_duration_milliseconds{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101",quantile="0.75"} 140
edgehub_message_process_duration_milliseconds{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101",quantile="0.95"} 145
edgehub_message_process_duration_milliseconds{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101",quantile="0.99"} 233

edgehub_message_process_duration_milliseconds_sum{to="IoTHub",from="d101/Sender0",edgeDevice="d101"} 35969
edgehub_message_process_duration_milliseconds_count{to="IoTHub",from="d101/Sender0",edgeDevice="d101"} 81
edgehub_message_process_duration_milliseconds{to="IoTHub",from="d101/Sender0",edgeDevice="d101",quantile="0.5"} 138
edgehub_message_process_duration_milliseconds{to="IoTHub",from="d101/Sender0",edgeDevice="d101",quantile="0.75"} 147
edgehub_message_process_duration_milliseconds{to="IoTHub",from="d101/Sender0",edgeDevice="d101",quantile="0.95"} 2229
edgehub_message_process_duration_milliseconds{to="IoTHub",from="d101/Sender0",edgeDevice="d101",quantile="0.99"} 2729

# HELP edgehub_message_size_bytes 
# TYPE edgehub_message_size_bytes summary
edgehub_message_size_bytes_sum{edgeDevice="d101"} 27596
edgehub_message_size_bytes_count{edgeDevice="d101"} 82
edgehub_message_size_bytes{edgeDevice="d101",quantile="0.5"} 337
edgehub_message_size_bytes{edgeDevice="d101",quantile="0.75"} 337
edgehub_message_size_bytes{edgeDevice="d101",quantile="0.95"} 337
edgehub_message_size_bytes{edgeDevice="d101",quantile="0.99"} 337

# HELP edgehub_gettwin_duration_milliseconds 
# TYPE edgehub_gettwin_duration_milliseconds summary
edgehub_gettwin_duration_milliseconds_sum{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101"} 0.1515155
edgehub_gettwin_duration_milliseconds_count{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101"} 1
edgehub_gettwin_duration_milliseconds{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101",quantile="0.5"} 0.1515155
edgehub_gettwin_duration_milliseconds{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101",quantile="0.75"} 0.1515155
edgehub_gettwin_duration_milliseconds{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101",quantile="0.95"} 0.1515155
edgehub_gettwin_duration_milliseconds{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101",quantile="0.99"} 0.1515155

# HELP edgehub_message_send_duration_milliseconds 
# TYPE edgehub_message_send_duration_milliseconds summary
edgehub_message_send_duration_milliseconds_sum{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101"} 2.5499204
edgehub_message_send_duration_milliseconds_count{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101"} 81
edgehub_message_send_duration_milliseconds{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101",quantile="0.5"} 0.0309645
edgehub_message_send_duration_milliseconds{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101",quantile="0.75"} 0.0324292
edgehub_message_send_duration_milliseconds{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101",quantile="0.95"} 0.0350137
edgehub_message_send_duration_milliseconds{from="d101/Sender0",to="d101/Receiver0",edgeDevice="d101",quantile="0.99"} 0.0829133

edgehub_message_send_duration_milliseconds_sum{to="IoTHub",from="d101/Sender0",edgeDevice="d101"} 6.3626492
edgehub_message_send_duration_milliseconds_count{to="IoTHub",from="d101/Sender0",edgeDevice="d101"} 64
edgehub_message_send_duration_milliseconds{to="IoTHub",from="d101/Sender0",edgeDevice="d101",quantile="0.5"} 0.0585582
edgehub_message_send_duration_milliseconds{to="IoTHub",from="d101/Sender0",edgeDevice="d101",quantile="0.75"} 0.0594094
edgehub_message_send_duration_milliseconds{to="IoTHub",from="d101/Sender0",edgeDevice="d101",quantile="0.95"} 0.0677852
edgehub_message_send_duration_milliseconds{to="IoTHub",from="d101/Sender0",edgeDevice="d101",quantile="0.99"} 2.6312966

# HELP edgehub_reported_properties_update_duration_milliseconds 
# TYPE edgehub_reported_properties_update_duration_milliseconds summary
edgehub_reported_properties_update_duration_milliseconds_sum{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101"} 1.0222126
edgehub_reported_properties_update_duration_milliseconds_count{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101"} 5
edgehub_reported_properties_update_duration_milliseconds{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101",quantile="0.5"} 0.1568278
edgehub_reported_properties_update_duration_milliseconds{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101",quantile="0.75"} 0.1592095
edgehub_reported_properties_update_duration_milliseconds{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101",quantile="0.95"} 0.1592095
edgehub_reported_properties_update_duration_milliseconds{Target="IoTHub",id="d101/$edgeHub",edgeDevice="d101",quantile="0.99"} 0.1592095

myagley · 2019-06-10T17:23:32Z

A couple of comments:

we should make all labels snake_case to be consistent. It looks like it is a mix of snake_case and PascalCase
Is it possible to get more percentiles? Specifically p999 and p9999 would be interesting
Should target/to IoTHub be upstream or $upstream instead?
If the library supports descriptions, it would be nice if we can get them to show up in the HELP sections

philipktlin · 2019-06-12T14:28:25Z

edge-util/src/Microsoft.Azure.Devices.Edge.Util/Microsoft.Azure.Devices.Edge.Util.csproj

@@ -22,6 +22,8 @@

  <ItemGroup>
    <PackageReference Include="App.Metrics" Version="3.0.0" />
+    <PackageReference Include="App.Metrics.Formatters.Prometheus" Version="3.2.0-dev0002" />


does 3.2.0-dev0002 like a rc version? should we use 3.1.0 instead?

yeah, good point, maybe we should.

philipktlin · 2019-06-12T14:30:53Z

edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/IMetricsCounter.cs

+    {
+        void Increment(long amount);
+        void Decrement(long amount);
+        void Increment(long amount, Dictionary<string, string> tags);


what kind of tags will be used here? should it be modeled more specific?

Emm, I don't think so. This is intended to be a generic Metrics API for the Edge.

varunpuranik added 13 commits June 3, 2019 18:34

Remove existing metrics

6de2f49

Metrics updates

7162593

Add basic metrics structure

3cfa12e

Add sending and receiving messages metrics

ad048a1

Working changes

2055a52

Add support for AppMetrics metrics

ef87862

Refactor

2f35dab

Refactor MetricsListener

bcd9dbe

Fix build

4ed9d6f

Pass in URL

5d7d3e2

Fix config

cf7da30

Add metrics and cleanup

af71bf0

Fix metrics

7a08f60

varunpuranik force-pushed the ehmetrics5 branch from bd7a0f3 to 7a08f60 Compare June 4, 2019 01:40

varunpuranik marked this pull request as ready for review June 4, 2019 01:40

varunpuranik changed the title ~~Draft: Metrics support for EdgeHub~~ Metrics support for EdgeHub Jun 4, 2019

varunpuranik added 5 commits June 4, 2019 10:01

Merge branch 'master' into ehmetrics5

403187d

Use Meter instead of Counter

be5cfa5

Update meter names

fbf7a1c

Fix metric names

af0dd83

Cleanup

78b7f8f

myagley reviewed Jun 6, 2019

View reviewed changes

varunpuranik added 3 commits June 6, 2019 10:02

Twin Metrics

2a8c59f

Fix default port

f76490a

Fix soem metrics

d0519f2

varunpuranik added 2 commits June 6, 2019 16:25

Pass through edge device id to metrics as a tag

85a8217

Merge branch 'master' into ehmetrics5

3031a6e

Merge branch 'master' into ehmetrics5

70f70bf

varunpuranik added 2 commits June 10, 2019 11:04

Fix casing

4b48f2d

Fix tests

604a357

philipktlin reviewed Jun 12, 2019

View reviewed changes

varunpuranik mentioned this pull request Jun 17, 2019

Metrics Support for EdgeHub (take #2) #1347

Merged

varunpuranik closed this Jun 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metrics support for EdgeHub #1290

Metrics support for EdgeHub #1290

varunpuranik commented Jun 3, 2019 •

edited

Loading

myagley commented Jun 6, 2019

myagley Jun 6, 2019

varunpuranik Jun 6, 2019

myagley Jun 6, 2019

myagley Jun 6, 2019

myagley Jun 6, 2019

varunpuranik Jun 6, 2019

myagley Jun 6, 2019

varunpuranik Jun 6, 2019

myagley Jun 6, 2019

myagley Jun 6, 2019

myagley Jun 6, 2019

varunpuranik Jun 6, 2019

myagley Jun 6, 2019

myagley Jun 6, 2019

varunpuranik Jun 6, 2019

myagley Jun 6, 2019

varunpuranik Jun 6, 2019

myagley Jun 6, 2019

varunpuranik Jun 6, 2019

varunpuranik commented Jun 6, 2019 •

edited by myagley

Loading

myagley commented Jun 10, 2019

philipktlin Jun 12, 2019

varunpuranik Jun 12, 2019

philipktlin Jun 12, 2019

varunpuranik Jun 12, 2019

Metrics support for EdgeHub #1290

Metrics support for EdgeHub #1290

Conversation

varunpuranik commented Jun 3, 2019 • edited Loading

myagley commented Jun 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

varunpuranik commented Jun 6, 2019 • edited by myagley Loading

myagley commented Jun 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

varunpuranik commented Jun 3, 2019 •

edited

Loading

varunpuranik commented Jun 6, 2019 •

edited by myagley

Loading