Implement QoS application monitoring #346
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pudelkoM
commented
Oct 5, 2021
|
pudelkoM
force-pushed
the
qos-measurements
branch
from
378692b
to
9579146
Compare
krsna1729
reviewed
Oct 8, 2021
|
when editing protobuf, run make pb to locally generate the go bindings |
pudelkoM
force-pushed
the
qos-measurements
branch
4 times, most recently
from
946c75e
to
a1ad7a5
Compare
grafana dashboard 1 json
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"gnetId": null,
"graphTooltip": 0,
"id": 2,
"iteration": 1634620612878,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "bps"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 0
},
"id": 2,
"options": {
"legend": {
"calcs": [],
"displayMode": "hidden",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"exemplar": true,
"expr": "upf_session_throughput_bps{fseid=~\"$fseid\", pdr=~\"$pdr\"}",
"format": "time_series",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "UPF Session Throughput bps",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "pps"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 9
},
"id": 3,
"options": {
"legend": {
"calcs": [],
"displayMode": "hidden",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"exemplar": true,
"expr": "upf_session_throughput_pps{fseid=~\"$fseid\", pdr=~\"$pdr\"}",
"format": "time_series",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "UPF Session Throughput pps",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 18
},
"id": 4,
"options": {
"legend": {
"calcs": [],
"displayMode": "hidden",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"exemplar": true,
"expr": "upf_session_drop_rate{fseid=~\"$fseid\", pdr=~\"$pdr\"}",
"format": "time_series",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "UPF Session Drop rate",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "ns"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 27
},
"id": 6,
"options": {
"legend": {
"calcs": [],
"displayMode": "hidden",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"exemplar": true,
"expr": "upf_session_latency_ns{fseid=~\"$fseid\", pdr=~\"$pdr\", quantile=~\"$quantile\"}",
"format": "time_series",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "UPF Session Latency",
"type": "timeseries"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "ns"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 36
},
"id": 5,
"options": {
"legend": {
"calcs": [],
"displayMode": "hidden",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"exemplar": true,
"expr": "upf_session_observation_duration_ns{fseid=~\"$fseid\", pdr=~\"$pdr\"}",
"format": "time_series",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "UPF Session Observation Duration",
"type": "timeseries"
}
],
"refresh": false,
"schemaVersion": 31,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"allValue": ".+",
"current": {
"selected": true,
"text": [
"All"
],
"value": [
"$__all"
]
},
"datasource": "Prometheus",
"definition": "label_values(fseid)",
"description": null,
"error": null,
"hide": 0,
"includeAll": true,
"label": null,
"multi": true,
"name": "fseid",
"options": [],
"query": {
"query": "label_values(fseid)",
"refId": "StandardVariableQuery"
},
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 3,
"type": "query"
},
{
"allValue": ".+",
"current": {
"selected": true,
"text": [
"All"
],
"value": [
"$__all"
]
},
"datasource": "Prometheus",
"definition": "label_values(pdr)",
"description": null,
"error": null,
"hide": 0,
"includeAll": true,
"label": null,
"multi": true,
"name": "pdr",
"options": [],
"query": {
"query": "label_values(pdr)",
"refId": "StandardVariableQuery"
},
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 3,
"type": "query"
},
{
"allValue": ".+",
"current": {
"selected": false,
"text": [
"All"
],
"value": [
"$__all"
]
},
"datasource": "Prometheus",
"definition": "label_values(upf_session_latency_ns, quantile)",
"description": null,
"error": null,
"hide": 0,
"includeAll": true,
"label": null,
"multi": true,
"name": "quantile",
"options": [],
"query": {
"query": "label_values(upf_session_latency_ns, quantile)",
"refId": "StandardVariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 3,
"type": "query"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "UPF Session Metrics",
"uid": "DwMjm2d7z",
"version": 14
}grafana dashboard 2 json
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"gnetId": null,
"graphTooltip": 0,
"id": 3,
"iteration": 1634644669009,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "continuous-GrYlRd"
},
"mappings": [],
"max": 20000000000,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "bps"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 6,
"x": 0,
"y": 0
},
"id": 2,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "8.2.1",
"targets": [
{
"exemplar": true,
"expr": "sum(upf_session_throughput_bps{fseid=~\"$fseid\", pdr=~\"$pdr\"})",
"format": "time_series",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "UPF Session Bit Rate",
"type": "stat"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "continuous-GrYlRd"
},
"mappings": [],
"max": 2000000,
"min": 0,
"thresholds": {
"mode": "percentage",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "pps"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 6,
"x": 6,
"y": 0
},
"id": 3,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "8.2.1",
"targets": [
{
"exemplar": true,
"expr": "sum(upf_session_throughput_pps{fseid=~\"$fseid\", pdr=~\"$pdr\"})",
"format": "time_series",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "UPF Session Packet Rate",
"type": "stat"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 6,
"x": 12,
"y": 0
},
"id": 4,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "8.2.1",
"targets": [
{
"exemplar": true,
"expr": "avg(upf_session_drop_rate{fseid=~\"$fseid\", pdr=~\"$pdr\"})",
"format": "time_series",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "UPF Session Drop Rate",
"type": "stat"
},
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "ns"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 6,
"x": 18,
"y": 0
},
"id": 5,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "8.2.1",
"targets": [
{
"exemplar": true,
"expr": "avg(upf_session_observation_duration_ns{fseid=~\"$fseid\", pdr=~\"$pdr\"})",
"format": "time_series",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "Observation Duration",
"type": "stat"
},
{
"cards": {
"cardPadding": null,
"cardRound": null
},
"color": {
"cardColor": "#b4ff00",
"colorScale": "sqrt",
"colorScheme": "interpolateOranges",
"exponent": 0.5,
"mode": "spectrum"
},
"dataFormat": "timeseries",
"datasource": "Prometheus",
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 9
},
"heatmap": {},
"hideZeroBuckets": false,
"highlightCards": true,
"id": 6,
"legend": {
"show": false
},
"reverseYBuckets": false,
"targets": [
{
"exemplar": true,
"expr": "upf_session_latency_ns{fseid=~\"$fseid\", pdr=~\"$pdr\", quantile=~\"$quantile\"}",
"format": "time_series",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "UPF Session Latency",
"tooltip": {
"show": true,
"showHistogram": false
},
"type": "heatmap",
"xAxis": {
"show": true
},
"xBucketNumber": null,
"xBucketSize": null,
"yAxis": {
"decimals": null,
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true,
"splitFactor": null
},
"yBucketBound": "auto",
"yBucketNumber": null,
"yBucketSize": null
}
],
"refresh": "1m",
"schemaVersion": 31,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"allValue": ".+",
"current": {
"selected": true,
"text": [
"All"
],
"value": [
"$__all"
]
},
"datasource": "Prometheus",
"definition": "label_values(fseid)",
"description": null,
"error": null,
"hide": 0,
"includeAll": true,
"label": null,
"multi": true,
"name": "fseid",
"options": [],
"query": {
"query": "label_values(fseid)",
"refId": "StandardVariableQuery"
},
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 3,
"type": "query"
},
{
"allValue": ".+",
"current": {
"selected": true,
"text": [
"All"
],
"value": [
"$__all"
]
},
"datasource": "Prometheus",
"definition": "label_values(pdr)",
"description": null,
"error": null,
"hide": 0,
"includeAll": true,
"label": null,
"multi": true,
"name": "pdr",
"options": [],
"query": {
"query": "label_values(pdr)",
"refId": "StandardVariableQuery"
},
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 3,
"type": "query"
},
{
"allValue": ".+",
"current": {
"selected": true,
"text": [
"All"
],
"value": [
"$__all"
]
},
"datasource": "Prometheus",
"definition": "label_values(upf_session_latency_ns, quantile)",
"description": null,
"error": null,
"hide": 0,
"includeAll": true,
"label": null,
"multi": true,
"name": "quantile",
"options": [],
"query": {
"query": "label_values(upf_session_latency_ns, quantile)",
"refId": "StandardVariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 3,
"type": "query"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "UPF Session Metrics Summary",
"uid": "cUFjx2dnz",
"version": 12
} |
|
Dashboard 1
Dashboard 2
|
diff --git a/pfcpiface/bess.go b/pfcpiface/bess.go
index a251007..054e9d0 100644
--- a/pfcpiface/bess.go
+++ b/pfcpiface/bess.go
@@ -666,6 +666,7 @@ func (b *bess) sim(u *upf, method string) {
precedence: 255,
+ pdrID: 1,
fseID: uint64(n3TEID + i),
ctrID: i,
farID: n3,
@@ -684,6 +685,7 @@ func (b *bess) sim(u *upf, method string) {
precedence: 1,
+ pdrID: 2,
fseID: uint64(n3TEID + i),
ctrID: i,
farID: n3,
@@ -705,6 +707,7 @@ func (b *bess) sim(u *upf, method string) {
precedence: 255,
+ pdrID: 3,
fseID: uint64(n3TEID + i),
ctrID: i,
farID: n6,
@@ -725,6 +728,7 @@ func (b *bess) sim(u *upf, method string) {
precedence: 1,
+ pdrID: 4,
fseID: uint64(n3TEID + i),
ctrID: i,
farID: n9, |
krsna1729
force-pushed
the
qos-measurements
branch
3 times, most recently
from
6d04e61
to
fccf3be
Compare
pudelkoM
commented
Oct 27, 2021
pfcpiface/bess.go
Outdated
Comment on lines
460
to
466
| if statsIn.Fseid != statsOut.Fseid || statsIn.Pdr != statsOut.Pdr { | ||
| continue | ||
| } |
There was a problem hiding this comment.
@krsna1729 We'd like to look up the right PDR here and use the IP addresses from it as a prometheus label. Unfortunately I have not yet found a way to access the pfcp session which keeps the slice of pdrs.
Any ideas?
There was a problem hiding this comment.
at the bess layer you wont find them. You can find it in PFCP layer https://github.com/omec-project/upf-epc/blob/master/pfcpiface/sessions.go#L18
Maybe you can use the setInfo call to setup what you need in bess struct pointing to the PFCPConn
https://github.com/omec-project/upf-epc/blob/master/pfcpiface/conn.go#L164
https://github.com/omec-project/upf-epc/blob/master/pfcpiface/bess.go#L71
They're now unique per module instance.
pudelkoM
commented
Nov 16, 2021
pudelkoM
commented
Nov 16, 2021
pudelkoM
force-pushed
the
qos-measurements
branch
2 times, most recently
from
b0ae1b8
to
264e676
Compare
pudelkoM
force-pushed
the
qos-measurements
branch
from
264e676
to
c4c1259
Compare
|
I think I addressed all immediate concerns. The entire module is now optional ( From ONF side these changes are sufficient, any (e.g. performance) improvements or generalization efforts can be done when needed. |
ccascone
previously approved these changes
Nov 22, 2021
krsna1729
reviewed
Nov 23, 2021
|
squash the commits |
|
Make necessary updates to the dashboards to get them working with the latest changes in the prometheus code |
The change to report counters instead of rates makes plotting bps/pps/drop_rate more complex. Unfortunately I don't know Prometheus well enough to make that change happen. There is also a pending discussion with the OPs team about the exact metrics and their format. |
pudelkoM
changed the title
ccascone
approved these changes
Nov 23, 2021
This was referenced Nov 28, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implementation of QoS monitoring as described here.
Steps to try:
command module qosMeasure read QosMeasureReadArg {'clear': False}docker logs -f bessExample output: