Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Metrics Support for EdgeHub (take #2) #1347

Merged
merged 21 commits into from
Jul 16, 2019
Merged

Metrics Support for EdgeHub (take #2) #1347

merged 21 commits into from
Jul 16, 2019

Conversation

varunpuranik
Copy link
Contributor

  • Remove existing metrics from EdgeHub codebase
  • Remove existing metrics infrastructure in Util
  • Add new generic metrics infrastructure in Util
  • Add support for prometheus style metrics
  • Add messaging metrics to EdgeHub
  • Addresses comments in previous PR - 1. support for 999th and 9999th percentile. and 2. Help text
    (Metrics support for EdgeHub #1290)

Sending out a separate PR since changes are significant.

@varunpuranik
Copy link
Contributor Author

Output of the metrics endpoint. Note that the dotnet metrics are added by the prometheus.net library by default, but seem useful.

# HELP process_private_memory_bytes Process private memory size
# TYPE process_private_memory_bytes gauge
process_private_memory_bytes 233353216
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2204064808960
# HELP dotnet_collection_count_total GC collection count
# TYPE dotnet_collection_count_total counter
dotnet_collection_count_total{generation="1"} 11
dotnet_collection_count_total{generation="0"} 49
dotnet_collection_count_total{generation="2"} 3
# HELP edge_hub_messages_sent_total Messages sent from edge hub
# TYPE edge_hub_messages_sent_total counter
edge_hub_messages_sent_total{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream"} 858
edge_hub_messages_sent_total{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0"} 859
# HELP edge_hub_message_process_duration_milliseconds Time taken to process message in EdgeHub
# TYPE edge_hub_message_process_duration_milliseconds summary
edge_hub_message_process_duration_milliseconds_sum{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream"} 73591
edge_hub_message_process_duration_milliseconds_count{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream"} 859
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.5"} 69
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.9"} 111
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.95"} 112
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.99"} 229
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.999"} 229
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.9999"} 229
edge_hub_message_process_duration_milliseconds_sum{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0"} 49578
edge_hub_message_process_duration_milliseconds_count{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0"} 859
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.5"} 10
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.9"} 111
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.95"} 112
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.99"} 184
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.999"} 184
edge_hub_message_process_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.9999"} 184
# HELP edge_hub_message_send_duration_milliseconds Time taken to send a message
# TYPE edge_hub_message_send_duration_milliseconds summary
edge_hub_message_send_duration_milliseconds_sum{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream"} 52.1423082
edge_hub_message_send_duration_milliseconds_count{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream"} 857
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.5"} 0.0585898
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.9"} 0.0611099
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.95"} 0.0700567
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.99"} 0.1757748
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.999"} 0.1757748
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="upstream",quantile="0.9999"} 0.1757748
edge_hub_message_send_duration_milliseconds_sum{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0"} 3.1270468
edge_hub_message_send_duration_milliseconds_count{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0"} 859
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.5"} 0.0034236
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.9"} 0.0052767
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.95"} 0.0057101
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.99"} 0.0656235
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.999"} 0.0656235
edge_hub_message_send_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",from="d101/Sender0",to="d101/Receiver0",quantile="0.9999"} 0.0656235
# HELP process_working_set_bytes Process working set
# TYPE process_working_set_bytes gauge
process_working_set_bytes 166100992
# HELP edge_hub_message_size_bytes Size of messages received by EdgeHub
# TYPE edge_hub_message_size_bytes summary
edge_hub_message_size_bytes_sum{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",id="d101/Sender0"} 290278
edge_hub_message_size_bytes_count{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",id="d101/Sender0"} 860
edge_hub_message_size_bytes{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",id="d101/Sender0",quantile="0.5"} 338
edge_hub_message_size_bytes{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",id="d101/Sender0",quantile="0.9"} 338
edge_hub_message_size_bytes{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",id="d101/Sender0",quantile="0.95"} 338
edge_hub_message_size_bytes{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",id="d101/Sender0",quantile="0.99"} 338
edge_hub_message_size_bytes{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",id="d101/Sender0",quantile="0.999"} 338
edge_hub_message_size_bytes{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",id="d101/Sender0",quantile="0.9999"} 338
# HELP edge_hub_reported_properties_total Reported properties update calls
# TYPE edge_hub_reported_properties_total counter
edge_hub_reported_properties_total{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",target="upstream",id="d101/$edgeHub"} 3
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 12.09375
# HELP edge_hub_reported_properties_update_duration_milliseconds Time taken to update reported properties
# TYPE edge_hub_reported_properties_update_duration_milliseconds summary
edge_hub_reported_properties_update_duration_milliseconds_sum{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",target="upstream",id="d101/$edgeHub"} 0.6519651
edge_hub_reported_properties_update_duration_milliseconds_count{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",target="upstream",id="d101/$edgeHub"} 3
edge_hub_reported_properties_update_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",target="upstream",id="d101/$edgeHub",quantile="0.5"} 0.1569458
edge_hub_reported_properties_update_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",target="upstream",id="d101/$edgeHub",quantile="0.9"} 0.1642532
edge_hub_reported_properties_update_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",target="upstream",id="d101/$edgeHub",quantile="0.95"} 0.1642532
edge_hub_reported_properties_update_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",target="upstream",id="d101/$edgeHub",quantile="0.99"} 0.1642532
edge_hub_reported_properties_update_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",target="upstream",id="d101/$edgeHub",quantile="0.999"} 0.1642532
edge_hub_reported_properties_update_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",target="upstream",id="d101/$edgeHub",quantile="0.9999"} 0.1642532
# HELP process_open_handles Number of open handles
# TYPE process_open_handles gauge
process_open_handles 1006
# HELP process_num_threads Total number of threads
# TYPE process_num_threads gauge
process_num_threads 76
# HELP edge_hub_messages_received_total Number of messages received from client
# TYPE edge_hub_messages_received_total counter
edge_hub_messages_received_total{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",protocol="mqtt",id="d101/Sender0"} 860
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1560797521.71009
# HELP edge_hub_get_twin_total Get twin calls
# TYPE edge_hub_get_twin_total counter
edge_hub_get_twin_total{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",source="upstream",id="d101/$edgeHub"} 1
# HELP dotnet_total_memory_bytes Total known allocated memory
# TYPE dotnet_total_memory_bytes gauge
dotnet_total_memory_bytes 177090616
# HELP edge_hub_gettwin_duration_milliseconds Time taken to get twin
# TYPE edge_hub_gettwin_duration_milliseconds summary
edge_hub_gettwin_duration_milliseconds_sum{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",source="upstream",id="d101/$edgeHub"} 0.1518479
edge_hub_gettwin_duration_milliseconds_count{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",source="upstream",id="d101/$edgeHub"} 1
edge_hub_gettwin_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",source="upstream",id="d101/$edgeHub",quantile="0.5"} 0.1518479
edge_hub_gettwin_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",source="upstream",id="d101/$edgeHub",quantile="0.9"} 0.1518479
edge_hub_gettwin_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",source="upstream",id="d101/$edgeHub",quantile="0.95"} 0.1518479
edge_hub_gettwin_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",source="upstream",id="d101/$edgeHub",quantile="0.99"} 0.1518479
edge_hub_gettwin_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",source="upstream",id="d101/$edgeHub",quantile="0.999"} 0.1518479
edge_hub_gettwin_duration_milliseconds{iot_hub="OfflineTestHub1.azure-devices.net",edge_device="d101",source="upstream",id="d101/$edgeHub",quantile="0.9999"} 0.1518479

myagley
myagley requested changes Jun 20, 2019
View reviewed changes
edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs Outdated Show resolved Hide resolved
edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs Outdated Show resolved Hide resolved
edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/BaseMetric.cs Show resolved Hide resolved
edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/IEdgeMetrics.cs Outdated Show resolved Hide resolved
edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/MetricsConstants.cs Outdated Show resolved Hide resolved
edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/MetricsConstants.cs Outdated Show resolved Hide resolved
edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/MetricsListener.cs Show resolved Hide resolved
edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/appMetrics/MetricsTimer.cs Show resolved Hide resolved
edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/prometheus.net/MetricsTimer.cs Show resolved Hide resolved
@myagley
Copy link
Contributor

myagley commented Jun 20, 2019

Lets definitely keep the default process level metrics. Lots of good stuff in there.

varunpuranik and others added 6 commits June 20, 2019 10:03
@varunpuranik @myagley
Update edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/Cloud…
df132a0 
…Proxy.cs

Co-Authored-By: Mike Yagley <myagley@gmail.com>
@varunpuranik @myagley
Update edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/Devi…
d918ff4 
…ceMessageHandler.cs

Co-Authored-By: Mike Yagley <myagley@gmail.com>
@varunpuranik @myagley
Update edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/IEdgeM…
e1caf56 
…etrics.cs

Co-Authored-By: Mike Yagley <myagley@gmail.com>
@varunpuranik @myagley
Update edge-util/src/Microsoft.Azure.Devices.Edge.Util/metrics/Metric…
b0431a1 
…sConstants.cs

Co-Authored-By: Mike Yagley <myagley@gmail.com>
@varunpuranik
Merge branch 'master' into ehmetrics10
94c8e71
@varunpuranik
Convert metrics duration to seconds
87aaddb
myagley
myagley approved these changes Jul 16, 2019
View reviewed changes
edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs
static readonly IMetricsHistogram MessagesProcessLatency = Util.Metrics.Metrics.Instance.CreateHistogram(
"message_process_duration_milliseconds",
static readonly IMetricsDuration MessagesProcessLatency = Util.Metrics.Metrics.Instance.CreateDuration(
"message_process_duration",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

message_process_duration_seconds?

@varunpuranik varunpuranik merged commit f93c6f3 into Azure:master Jul 16, 2019
@varunpuranik varunpuranik deleted the ehmetrics10 branch July 16, 2019 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@myagley myagley myagley approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@varunpuranik @myagley