Dynamic Bandwidth Monitor; leak detection method implemented in a real-time data historian
Switch branches/tags
v1.30.6738.17066+55afcef v1.24.6738.16805+59bd936 v1.24.6738.14079+7169e7a v1.24.6737.11513+29a3235 v1.24.6736.28304+066f76f v1.24.6736.25891+dc2b16b v1.24.6736.23729+dfd0885 v1.24.6736.23527+f370230 v1.24.6736.21659+94d6e7e v1.24.6736.13604+a234217 v1.22.6734.33741+3e929db v1.22.6733.23495+1cac28b v1.21.6733.10389+f99087f v1.21.6731.17083+9881c64 v1.21.6731.15667+92512f2 v1.21.6731.14469+d76815f v1.21.6730.30836+7402387 v1.21.6729.16773+0d4c380 v1.21.6728.16194+daa3f9f v1.21.6728.15436+ea44bf4 v1.21.6728.9179+6badd37 v1.21.6727.31247+fc9d5ea v1.21.6726.25075+65bb4c7 v1.21.6726.11491+84c6d83 v1.21.6725.34411+890b1b8 v1.21.6725.18919+4d4db8b v1.21.6724.35262+fc53613 v1.21.6724.20985+ce4dcb4 v1.20.6722.12467+66bd90f v1.20.6718.17027+d29942e v1.20.6718.16040+79ee868 v1.20.6718.14923+802aa45 v1.20.6717.19884+8cc68b0 v1.19.6716.21479+a44dc7b v1.19.6712.18974+2ed2df0 v1.19.6712.10485+1989b2a v1.19.6711.25514+114beeb v1.19.6711.25224+c1d3741 v1.19.6711.11088+3f24a17 v1.19.6711.10682+ac115d1 v1.19.6710.22848+4b7aa26 v1.19.6705.13794+93ae213 v1.19.6702.21168+e8d705b v1.19.6695.11602+8046fee v1.19.6690.21968+3331601 v1.19.6690.13756+3d5939c v1.19.6689.19725+5ba809e v1.19.6688.24800+0170efd v1.19.6687.22670+1687ec1 v1.19.6687.13917+db05e39 v1.18.6683.31331+6bdfc20 v1.18.6683.30420+8c277c3 v1.18.6683.23647+5365a79 v1.18.6683.16920+469dc3a v1.18.6683.12109+d26dd00 v1.18.6681.14254+eb0f190 v1.18.6679.20766+b014cbd v1.18.6677.22708+e3ea069 v1.18.6674.14837+a1558b3 v1.18.6673.20932+5360a75 v1.18.6673.18766+cc3a1c5 v1.17.6663.13670+d204be3 v1.17.6640.17310+4740a07 v1.17.6635.21239+605fe21 v1.16.6613.15700+1188131 v1.16.6584.16998+5e1e165 v1.16.6580.11444+c3403e0 v1.16.6578.25325+36bbee7 v1.16.6578.18794+9e9bd5f v1.15.6578.15554+7393598 v1.15.6577.21786+f4a63d0 v1.15.6577.11937+b55326c v1.15.6573.32021+68bece6 v1.14.6573.31966+8129521 v1.14.6573.31930+43be832 v1.14.6573.31892+ca48026 v1.14.6572.14011+fd58daa v1.14.6571.18332+3935452 v1.14.6571.13090+384f436 v1.14.6571.12820+3a74fc7 v1.14.6571.12771+904f1d4 v1.14.6565.21502+b982dde v1.14.6565.21171+511a358 v1.14.6563.15112+7ca7b26 v1.14.6563.14167+fe28b7f v1.14.6563.12912+fc5cab0 v1.14.6563.12081+d95b633 v1.14.6562.24880+b17ab3c v1.14.6562.21124+4b3585a v1.14.6562.17136+c231b84 v1.14.6562.14248+28228b9 v1.14.6562.13956+77b81a5 v1.14.6561.34207+2b1f9d5 v1.14.6561.33756+eacb34b v1.14.6561.25303+2ac3885 v1.14.6560.31417+fbfe09b v1.12.24.36009+1dd5f2c v1.12.24.34745+9010674 v1.12.24.33999+afa5c23 v1.12.20.32964+1029815
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
img
samples
src
.gitignore
AUTHORS
CODE_OF_CONDUCT.md
CONTRIBUTORS
LICENSE
README.md
appveyor.yml
build.bat
build.sh

README.md

Logo

DBM

Dynamic Bandwidth Monitor
Leak detection method implemented in a real-time data historian
Copyright (C) 2014-2018 J.H. Fitié, Vitens N.V.

Continuous integration

Build status Downloads
Build status 📦 Latest release

Description

Water company Vitens has created a demonstration site called the Vitens Innovation Playground (VIP), in which new technologies and methodologies are developed, tested, and demonstrated. The projects conducted in the demonstration site can be categorized into one of four themes: energy optimization, real-time leak detection, online water quality monitoring, and customer interaction. In the real-time leak detection theme, a method for leak detection based on statistical demand forecasting was developed.

Using historical demand patterns and statistical methods - such as median absolute deviation, linear regression, sample variance, and exponential moving averages - real-time values can be compared to a forecast demand pattern and checked to be within calculated bandwidths. The method was implemented in Vitens' realtime data historian, continuously comparing measured demand values to be within operational bounds.

One of the advantages of this method is that it doesn't require manual configuration or training sets. Next to leak detection, unmeasured supply between areas and unscheduled plant shutdowns were also detected. The method was found to be such a success within the company, that it was implemented in an operational dashboard and is now used in day-to-day operations.

Keywords

Real-time, leak detection, demand forecasting, demand patterns, operational dashboard

Samples

Sample 1 - Forecast

Sample 1

In this example, two days before and after the current day are shown. For historic values, the measured data (black) is shown along with the forecast value (red). The upper and lower control limits (gray) were not crossed, so the DBM factor value (blue) equals zero. For future values, the forecast is shown along with the upper and lower control limits. Reliable forecasts can be made for at least seven days in advance.

Sample 2 - Event

Sample 2

In this example, an event causes the measured value (black) to cross the upper control limit (gray). The DBM factor value (blue) is greater than one during this time (calculated as (measured value - forecast value)/(upper control limit - forecast value)).

Sample 3 - Suppressed event (correlation)

Sample 3a Sample 3b

In this example, an event causes the measured value (black) to cross the upper and lower control limits (gray). Because the pattern is checked against a similar pattern which has a comparable relative forecast error (calculated as (forecast value / measured value) - 1), the event is suppressed. The DBM factor value is reset to zero during this time.

Sample 4 - Suppressed event (anticorrelation)

Sample 4a Sample 4b

In this example, an event causes the measured value (black) to cross the lower control limit (gray). Because the pattern is checked against a similar, adjacent, pattern which has a comparable, but inverted, absolute forecast error (calculated as forecast value - measured value), the event is suppressed. The DBM factor value is reset to zero during this time.

Program information

Requirements

Priority Requirement
Mandatory Microsoft .NET Framework
Optional OSIsoft PI Asset Framework Software Development Kit (PI AF SDK)

Drivers

DBM uses drivers to read information from a source of data. The following drivers are included:

Driver Description Identifier (Point) Remarks
DBMPointDriverCSV.vb Driver for CSV files (timestamp,value). String (CSV filename) Data interval must be the same as the CalculationInterval parameter.
DBMPointDriverOSIsoftPIAF.vb Driver for OSIsoft PI Asset Framework. OSIsoft.AF.Asset.AFAttribute (PI AF attribute) Used by PI AF Data Reference DBMDataRef.

Parameters

DBM can be configured using several parameters. The values for these parameters can be changed at runtime in the Vitens.DynamicBandwidthMonitor.DBMParameters class.

Parameter Default value Units Description
CalculationInterval 300 seconds Time interval at which the calculation is run.
ComparePatterns 12 weeks Number of weeks to look back to forecast the current value and control limits.
EMAPreviousPeriods 5 intervals Number of previous intervals used to smooth the data.
OutlierCI 0.99 ratio Confidence interval used for removing outliers.
BandwidthCI 0.99 ratio Confidence interval used for determining control limits.
CorrelationPreviousPeriods 23 intervals Number of previous intervals used to calculate forecast error correlation when an event is found.
CorrelationThreshold 0.83666 dimensionless Absolute correlation lower limit for detecting (anti)correlation.
RegressionAngleRange 18.435 degrees Regression angle range (around -45/+45 degrees) required when suppressing based on (anti)correlation.

DBMTester

DBMTester is a command line utility that can be used to quickly calculate DBM results using the CSV driver. The following arguments are available:

Argument Count Description
-i= 1 Specifies the input point (CSV file).
-c= 0..n Adds a correlation point (CSV file).
-cs= 0..n Adds a correlation point (CSV file) from which the input point is subtracted.
-iv= 0..1 Changes the CalculationInterval parameter.
-p= 0..1 Changes the ComparePatterns parameter.
-ep= 0..1 Changes the EMAPreviousPeriods parameter.
-oi= 0..1 Changes the OutlierCI parameter.
-bi= 0..1 Changes the BandwidthCI parameter.
-cp= 0..1 Changes the CorrelationPreviousPeriods parameter.
-ct= 0..1 Changes the CorrelationThreshold parameter.
-ra= 0..1 Changes the RegressionAngleRange parameter.
-st= 1 Start timestamp for calculations.
-et= 0..1 End timestamp for calculations, all intervals in between are calculated.
-f= 0..1 Output format. local (default) for local formatting, intl for UTC time and international formatting (ISO 8601).

DBMDataRef

DBMDataRef is a custom OSIsoft PI Asset Framework data reference which integrates DBM with PI AF. The build script automatically registers the data reference and support assemblies when run on the PI AF server. The data reference uses the parent attribute as input and automatically uses attributes from sibling and parent elements based on the same template containing good data for correlation calculations, unless the NoCorrelation category is applied to the output attribute. The value returned from the DBM calculation is determined by the applied property/trait:

Property/trait Return value
None Factor
Target Measured value
Forecast Forecast value
Minimum Lower control limit (p = 0.9999)
LoLo Lower control limit (default)
Lo Lower control limit (p = 0.95)
Hi Upper control limit (p = 0.95)
HiHi Upper control limit (default)
Maximum Upper control limit (p = 0.9999)

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

About Vitens

Vitens is the largest drinking water company in The Netherlands. We deliver top quality drinking water to 5.6 million people and companies in the provinces Flevoland, Fryslan, Gelderland, Utrecht and Overijssel and some municipalities in Drenthe and Noord-Holland. Annually we deliver 350 million m3 water with 1,400 employees, 100 water treatment works and 49,000 kilometres of water mains.

One of our main focus points is using advanced water quality, quantity and hydraulics models to further improve and optimize our treatment and distribution processes.

Vitens

https://www.vitens.nl/