Skip to content
A feature-based procedure for detecting technical outliers in water-quality data from in situ sensors πŸ’¦πŸ’¦πŸ’¦πŸ’¦πŸ’¦πŸ’¦πŸ’¦πŸ’¦πŸ’¦β˜”πŸ’¦πŸ’¦πŸ’¦πŸ’¦πŸ’¦πŸ’¦πŸ’¦
R
Branch: master
Clone or download
Latest commit 9314385 Aug 27, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
R update description of data Aug 21, 2019
data added new data Sep 7, 2018
inst/Shiny-app/Oddwater_app update shiny app Aug 26, 2019
man update description of data Aug 21, 2019
.Rbuildignore added travis.yml Aug 9, 2018
.gitignore .DS_Store banished! May 18, 2019
.travis.yml edit travis file Aug 9, 2018
DESCRIPTION update package version Aug 27, 2019
NAMESPACE added NN-HD fucntion May 17, 2019
README.Rmd update readme Aug 27, 2019
README.md update readme Aug 27, 2019
oddwater.Rproj added package Aug 9, 2018

README.md

oddwater

Project Status: WIP ? Initial development is in progress, but there has not yet been a stable, usable release suitable for the public. Licence


minimal R version CRAN_Status_Badge packageversion DOI


Last-changedate

Build Status

oddwater

Outliers due to technical errors in water-quality data from in situ sensors can reduce data quality and have a direct impact on inference drawn from subsequent data analysis. Here, we proposed an automated framework that provides early detection of outliers in water-quality data from in situ sensors caused by technical issues. The framework was used first to identify the data features that differentiate outlying instances from typical behaviours. Then statistical transformations were applied to make the outlying instances stand out in transformed data space. Unsupervised outlier scoring techniques were then applied to the transformed data space and an approach based on extreme value theory was used to calculate a threshold for each potential outlier.

In this work we showed that our proposed framework, with carefully selected data transformation methods derived from data features, can greatly assist in increasing the performance of a range of existing outlier detection algorithms.

This package is still under development and this repository contains a development version of the R package oddwater.

Installation

You can install oddwater from github with:

# install.packages("devtools")
devtools::install_github("pridiltal/oddwater")

Illustrative Example

Data obtained from in situ sensors at Sandy Creek.

library(tidyverse)
library(oddwater)
data("data_sandy_anom")
head(data_sandy_anom)
#>         Timestamp Level   Cond   Tur label_Level label_Cond label_Tur
#> 1 12/03/2017 1:00 0.636 326.34 34.47           0          0         0
#> 2 12/03/2017 2:30 0.636 326.63 34.06           0          0         0
#> 3 12/03/2017 4:00 0.635 327.00 33.39           0          0         0
#> 4 12/03/2017 5:30 0.633 327.31 32.61           0          0         0
#> 5 12/03/2017 7:00 0.634 327.81 33.34           0          0         0
#> 6 12/03/2017 8:30 0.631 328.70 32.22           0          0         0
#>   type_Level type_Cond type_Tur
#> 1          0         0        0
#> 2          0         0        0
#> 3          0         0        0
#> 4          0         0        0
#> 5          0         0        0
#> 6          0         0        0

Shiny App

oddwater::explore_data()
You can’t perform that action at this time.