CRAN Task View: Anomaly Detection with R
Maintainer: Priyanga Dilini Talagala, Rob J. Hyndman
Contact: Dilini.Talagala at monash.edu
This CRAN task view contains a list of packages that can be used for anomaly detection. Anomaly detection problems have many different facets and the detection techniques can be highly influenced by the way we define anomalies, the type of input data to the algorithm, the expected output, etc. This leads to wide variations in problem formulations, which need to be addressed through different analytical approaches.
Anomalies are often mentioned under several alternative names such as outliers, novelty, odd values, extreme values, faults, aberration in different application domains. These variants are also considered for this task view.
The development of this task view is fairly new and still in its early stages and therefore subject to changes. Please send suggestions for additions and extensions for this task view to the task view maintainer.
Univariate Outlier Detection
- Univariate outlier detection methods focus on values in a single feature space. Package univOutl includes various methods for detecting univariate outliers, e.g. the Hidiroglou-Berthelot method. Methods to deal with skewed distribution are also included in this package.
- The dixonTest package provides Dixon's ratio test for outlier detection in small and normally distributed samples.
- Univariate outliers detection is also supported by
outlier()function in GmAMisc which implements three different methods (mean-base, median-based, boxplot-based).
- The hotspots package supports univariate outlier detection by identifying values that are disproportionately high based on both the deviance of any given value from a statistical distribution and its similarity to other values.
- The outliers package provides a collection of tests commonly used for identifying outliers . For most functions the input is a numeric vector. If argument is a data frame, then outlier is calculated for each column by sapply. The same behavior is applied by apply when the matrix is given.
- The extremevalues package offers outlier detection and plot functions for univariate data. In this work a value in the data is an outlier when it is unlikely to be drawn from the estimated distribution.
- The funModeling package provides tools for outlier detection using top/bottom X%, Tukey’s boxplot definition and Hampel’s method.
- The alphaOutlier package provides Alpha-Outlier regions (as proposed by Davies and Gather (1993)) for well-known probability distributions.
Multivariate Outlier Detection
- Under multivariate, high-dimensional or multidimensional scenario, where the focus is on n (>2) - dimensional space, all attributes might be of same type or might be a mixture of different types such as categorical or numerical, which has a direct impact on the implementation and scope of the algorithm. The problems of anomaly detection in high-dimensional data are threefold, involving detection of: (a) global anomalies, (b) local anomalies and (c) micro clusters or clusters of anomalies. Global anomalies are very different from the dense area with respect to their attributes. In contrast, a local anomaly is only an anomaly when it is distinct from, and compared with, its local neighbourhood. Micro clusters or clusters of anomalies may cause masking problems.
Multivariate Outlier Detection: Density-based outlier detection
- The DDoutlier package provides a wide variety of distance- and density-based outlier detection functions mainly focusing local outliers in high-dimensional data.
- The OutlierDetection package provides different implementations for outlier detection namely model based, distance based, dispersion based, depth based and density based. This package provides labelling of observations as outliers and outlierliness of each outlier. For univariate, bivariate and trivariate data, visualization is also provided.
- Local Outlier Factor (LOF) is an algorithm for detecting anomalous
data points by measuring the local deviation of a given data point
with respect to its neighbours. This algorithm with some variations
is supported by many packages. The
DescTools package provides
functions for outlier detection using LOF and Tukey’s boxplot
GLOSHin package dbscan provide density based anomaly detection methods using a kd-tree to speed up kNN search. Parallel implementation of LOF which uses multiple CPUs to significantly speed up the LOF computation for large datasets is available in Rlof package. Package bigutilsr provides utility functions for outlier detection in large-scale data. It includes LOF and outlier detection method based on departure from histogram.
- The SMLoutliers package provides an implementation of the Local Correlation Integral method (Lof: Identifying density-based local outliers) for outlier detection in multivariate data which consists of numeric values.
- The ldbod package provides flexible functions for computing local density-based outlier scores. It allows for subsampling of input data or a user specified reference data set to compute outlier scores against, so both unsupervised and semi-supervised outlier detection can be done.
- The kernlab package provides kernel-based machine learning methods including one-class Support Vector Machines for novelty detection.
- The amelie package implements anomaly detection as binary classification for multivariate
- The estimated density ratio function in densratio package can be used in many applications such as anomaly detection, change-point detection, covariate shift adaptation.
Multivariate Outlier Detection: Distance-based outlier detection
- The HDoutliers package provides an implementation of an algorithm for univariate and multivariate outlier detection that can handle data with a mixed categorical and continuous variables and outlier masking problem.
- The stray package implements an algorithm for detecting anomalies in high-dimensional data that addresses the limitations of 'HDoutliers' algorithm. An approach based on extreme value theory is used for the anomalous threshold calculation.
- The mvoutlier package provides multivariate outlier detection based on robust methods.
- The Routliers package provides robust methods to detect univariate (Median Absolute Deviation method) and multivariate outliers (Mahalanobis-Minimum Covariance Determinant method).
- The modi package implements Mahalanobis distance or depth-based algorithms for multivariate outlier detection in the presence of missing values (incomplete survey data).
- The CerioliOutlierDetection package implements the iterated RMCD method of Cerioli (2010) for multivariate outlier detection via robust Mahalanobis distances.
- The rrcovHD package performs outlier identification using robust multivariate methods based on robust mahalanobis distances and principal component analysis.
dm.mahalanobisin DJL package implements Mahalanobis distance measure for outlier detection. In addition to the basic distance measure, boxplots are provided with potential outlier(s) to give an insight into the early stage of data cleansing task.
Multivariate Outlier Detection: Clustering-based outlier detection
- The kmodR package presents a unified approach for simultaneously clustering and discovering outliers in high dimensional data. Their approach is formalized as a generalization of the k-MEANS problem.
- The CrossClustering package implements a partial clustering algorithm that combines the Ward's minimum variance and Complete Linkage algorithms, providing automatic estimation of a suitable number of clusters and identification of outlier elements.
- The DMwR2 package uses hierarchical clustering to obtain a ranking of outlierness for a set of cases. The ranking is obtained on the basis of the path each case follows within the merging steps of a agglomerative hierarchical clustering method.
Multivariate Outlier Detection: Angle-based outlier detection
- The abodOutlier package performs angle-based outlier detection on high dimensional data. A complete, a randomized and a knn based methods are available.
- The HighDimOut package provides three high-dimensional outlier detection algorithms (angle-based, subspace based, feature bagging-based) and an outlier unification scheme.
Multivariate Outlier Detection: Decision tree based approaches
- Explainable outlier detection method through decision tree conditioning is facilitated by outliertree package .
- The isotree package provides fast and multi-threaded implementation of Extended Isolation Forest, Fair-Cut Forest, SCiForest (a.k.a. Split-Criterion iForest), and regular Isolation Forest, for isolation-based outlier detection, clustered outlier detection, distance or similarity approximation, and imputation of missing values based on random or guided decision tree splitting. It also supports categorical data.
- The outForest package provides a random forest based implementation for multivariate outlier detection. In this method each numeric variable is regressed onto all other variables by a random forest. If the scaled absolute difference between observed value and out-of-bag prediction of the corresponding random forest is suspiciously large, then a value is considered an outlier.
Multivariate Outlier Detection: Other approaches
- A set of algorithms for detection of outliers based on frequent pattern mining is available in fpmoutliers package. Such algorithms follow the paradigm: if an instance contains more frequent patterns,it means that this data instance is unlikely to be an anomaly.
- The ICSOutlier package performs multivariate outlier detection using invariant coordinates and offers different methods to choose the appropriate components. The current implementation targets data sets with only a small percentage of outliers but future extensions are under preparation.
- The sGMRFmix package provides an anomaly detection method for multivariate noisy sensor data using sparse Gaussian Markov random field mixtures. It can compute variable-wise anomaly scores.
- Artificial neural networks for anomaly detection is implemented in ANN2 package.
- The probout package estimates unsupervised outlier probabilities for multivariate numeric
- The mrfDepth package provides tools to compute depth measures and implementations of related tasks such as outlier detection, data exploration and classification of multivariate, regression and functional data.
- The evtclass package provides two classifiers for open set recognition and novelty detection based on extreme value theory.
- The dlookr package provides a collection of tools that support data diagnosis, exploration, and transformation. Data diagnostics provides information and visualization of missing values and outliers and unique and negative values to understand the distribution and quality of data.
- The RaPKod package implements a kernel method that performs online outlier detection through random low dimensional projections in a kernel space on the basis of a reference set of non-outliers.
- The FastHCS package implements robust algorithm for principal component analysis and thereby provide robust PCA modelling and associated outlier detection and diagnostic tools for high-dimensional data. PCA based outlier detection tools are also available via FactoInvestigate package.
- Cellwise outliers are entries in the data matrix which are substantially higher or lower than what could be expected based on the other cells in its column as well as the other cells in its row, taking the relations between the columns into account. Package cellWise provides tools for detecting cellwise outliers and robust methods to analyze data which may contain them.
- The Projection Congruent Subset (PCS) is a method for finding multivariate outliers by searching for a subset which minimizes a criterion. PCS is supported by FastPCS package.
- The problems of anomaly detection for temporal data are 3-fold: (a) the detection of contextual anomalies (point anomalies) within a given series; (b) the detection of anomalous subsequences within a given series; and (c) the detection of anomalous series within a collection of series
- The trendsegmentR package performs the detection of point anomalies and linear trend changes for univariate time series by implementing the bottom-up unbalanced wavelet transformation.
- The anomaly package implements Collective And Point Anomaly (CAPA), Multi-Variate Collective And Point Anomaly (MVCAPA), and Proportion Adaptive Segment Selection (PASS) methods for the detection of anomalies in time series data.
- The anomalize package enables a
"tidy" workflow for detecting anomalies in data. The main functions
- The cbar package detect contextual anomalies in time-series data with Bayesian data analysis. It focuses on determining a normal range of target value, and provides simple-to-use functions to abstract the outcome.
detectIOfunctions in TSA package support detecting additive outlier and innovative outlier in time series data.
- The washeR package performs time series outlier detection using non parametric test. An input can be a data frame (grouped time series: phenomenon+date+group+values) or a vector (single time series)
- The tsoutliers package implements the Chen-Liu approach for detection of time series outliers such as innovational outliers, additive outliers, level shifts, temporary changes and seasonal level shifts.
- The seasonal package provides easy-to-use interface to X-13-ARIMA-SEATS, the seasonal adjustment software by the US Census Bureau. It offers full access to almost all options and outputs of X-13, including outlier detection.
- The npphen package implements basic and high-level functions for detection of anomalies in vector data (numerical series/ time series) and raster data (satellite derived products). Processing of very large raster files is supported.
- the ACA package offers an interactive function for the detection of abrupt change-points or aberrations in point series.
- The mmppr (Markov modulated Poisson process) package provides a framework for detecting anomalous events in time series of counts using an unsupervised learning approach.
- The surveillance package implements statistical methods for aberration detection in time series of counts, proportions and categorical data, as well as for the modeling of continuous-time point processes of epidemic phenomena. The package also contains several real-world data sets, the ability to simulate outbreak data, and to visualize the results of the monitoring in a temporal, spatial or spatio-temporal fashion.
- A set of online fault (anomaly) detectors for time series using prediction-based and window-based techniques are available via otsad package. It can handle both stationary and non-stationary environments.
- The SmartSifter package provides online unsupervised outlier detection methods using finite mixtures with discounting learning algorithms.
- The oddstream package implements an algorithm for early detection of anomalous series within a large collection of streaming time series data. The model uses time series features as inputs, and a density-based comparison to detect any significant changes in the distribution of the features.
- Spatial objects whose non-spatial attribute values are markedly different from those of their spatial neighbors are known as Spatial outliers or abnormal spatial patterns.
- The RWBP package detects spatial outliers using a Random Walk on Bipartite Graph.
- Enhanced False Discovery Rate (EFDR) is a tool to detect anomalies in an image. Package EFDR implements wavelet-based Enhanced FDR for detecting signals from complete or incomplete spatially aggregated data. The package also provides elementary tools to interpolate spatially irregular data onto a grid of the required size.
- The function
spatial.outlierin depth.plot package helps to identify multivariate spatial outlier within a p-variate data cloud or if any p-variate observation is an outlier with respect to a p-variate data cloud.
- Scan statistics are used to detect anomalous clusters in spatial or space-time data. The scanstatistics package provides functions for detection of anomalous space-time clusters using the scan statistics methodology. It focuses on prospective surveillance of data streams, scanning for clusters with ongoing anomalies.
- The solitude package provides an implementation of Isolation forest which detects anomalies purely based on the concept of isolation without employing any distance or density measure.
- Functions for error detection and correction in point data quality datasets that are used in species distribution modelling are available via biogeo package.
- The CoordinateCleaner package provides functions for flagging of common spatial and temporal outliers (errors) in biological and paleontological collection data, for the use in conservation, ecology and paleontology.
foutliers()function from rainbow package provides functional outlier detection methods. Bagplots and boxplots for functional data can also be used to identify outliers, which have either the lowest depth (distance from the centre) or the lowest density, respectively.
- The adamethods package provides a collection of several algorithms to obtain archetypoids with small and large databases and with both classical multivariate data and functional data (univariate and multivariate). Some of these algorithms also allow to detect anomalies.
shape.fd.outliersfunction in ddalpha package detects functional outliers of first three orders, based on the order extended integrated depth for functional data.
- The fda.usc package provides tools for outlier detection in functional data (atypical curves detection) using different approaches such as likelihood ratio test, depth measures, quantiles of the bootstrap samples.
- The fdasrvf package supports outlier detection in functional data using the square-root velocity framework which allows for elastic analysis of functional data through phase and amplitude separation.
Visualization of Anomalies
- The OutliersO3 package provides tools to aid in the display and understanding of patterns of multivariate outliers. It uses the results of identifying outliers for every possible combination of dataset variables to provide insight into why particular cases are outliers.
- The Morpho package provides a collection of tools for Geometric Morphometrics and mesh processing. Apart from the core functions it provides a graphical interface to find outliers and/or to switch mislabeled landmarks.
- The StatDA package provides visualization tools to locate outliers in environmental data.
Pre-processing Methods for Anomaly Detection
- The preprocomb package provides an S4 framework for creating and evaluating preprocessing combinations for classification, clustering and outlier detection.
- The dobin package provides dimension reduction technique for outlier detection using neighbours, constructs a set of basis vectors for outlier detection. It brings outliers to the fore-front using fewer basis vectors.
Specific Application Fields
- The precintcon package contains functions to analyze the precipitation intensity, concentration and anomaly.
- The survBootOutliers package provides concordance based bootstrap methods for outlier detection in survival analysis.
- The pcadapt package provides methods to detect genetic markers involved in biological adaptation using statistical tools based on Principal Component Analysis.
- The rgr package supports exploratory data analysis with applied geochemical data, with special application to the estimation of background ranges and identification of anomalies to support mineral exploration and environmental studies.
- The NMAoutlier package implements the forward search algorithm for the detection of outlying studies (studies with extreme results) in network meta-analysis.
- The KRIS package provides useful functions which are needed for bioinformatic analysis including detection of rough structures and outliers using unsupervised clustering.
- The dave package provides a collection of functions for data analysis in vegetation ecology including outlier detection using nearest neighbour distances.
- The MALDIrppa package provides methods for quality control and robust pre-processing and analysis of MALDI mass spectrometry data.
- The MIPHENO package contains functions to carry out processing of high throughput data analysis and detection of putative hits/mutants.
- The OutlierDM package provides functions to detect outlying values such as genes, peptides or samples for multi-replicated high-throughput high-dimensional data.
- The OutlierDC package implements algorithms to detect outliers based on quantile regression for censored survival data.
- The qpcR package implements methods for kinetic outlier detection (KOD) in real-time polymerase chain reaction (qPCR).
- The referenceIntervals package provides a collection of tools including outlier detcetion to allow the medical professional to calculate appropriate reference ranges (intervals) with confidence intervals around the limits for diagnostic purposes.
- The Hampel filter is a robust outlier detector using Median Absolute Deviation (MAD). The seismicRoll package provides fast rolling functions for seismology including outlier detection with a rolling Hampel Filter.
- The spikes package provides tool to detect election fraud from irregularities in vote-share distributions using re-sampled kernel density method.
- The wql package stands for `water quality' provides functions including anomaly detection to assist in the processing and exploration of data from environmental monitoring programs.
- The Grubbs‐Beck test is recommended by the federal guidelines for detection of low outliers in flood flow frequency computation in the United States. The MGBT computes the multiple Grubbs-Beck low-outlier test on positively distributed data and utilities for non-interpretive U.S. Geological Survey annual peak-stream flow data processing.
- The envoutliers package provides three semi-parametric methods for detection of outliers in environmental data based on kernel regression and subsequent analysis of smoothing residuals
- The rIP package supports detection of fraud in online surveys by tracing, scoring, and visualizing IP addresses
- The extremeIndex computes an index measuring the amount of information brought by forecasts for extreme events, subject to calibration. This index is originally designed for weather or climate forecasts, but it may be used in other forecasting contexts.
- The clampSeg package provides tool to identify and idealize flickering events in filtered ion channel recordings.
- The anomaly package contains lightcurve time series data from the Kepler telescope.
- Various high dimensional datasets are provided by mvoutlier package.
- The leri package finds and downloads Landscape Evaporative Response Index (LERI) data, then reads the data into R. The LERI product measures anomalies in actual evapotranspiration, to support drought monitoring and early warning systems.
- The waterData package imports U.S. Geological Survey (USGS) daily hydrologic data from USGS web services and provides functions to calculate and plot anomalies.
- The analytics package provides support for (among other functions) outlier detection in a fitted linear model.
- The nlr package include tools to detecting outliers in nonlinear regression.
- The CircOutlier package enables detection of outliers in circular-circular regression models, modifying its and estimating of models parameters.
- The Residual Congruent Subset (RCS) is a method for finding outliers in the regression setting. RCS is supported by FastRCS package.
- Package quokar provides quantile regression outlier diagnostics with K Left Out Analysis.
- The oclust package provides a function to detect and trim outliers in Gaussian mixture model based clustering using methods described in Clark and McNicholas (2019).
- The semdiag package implements outlier and leverage diagnostics for Structural equation modeling.
- The SeleMix package provides functions for detection of outliers and influential errors using a latent variable model. A mixture model (Gaussian contamination model) based on response(s) y and a depended set of covariates is fit to the data to quantify the impact of errors to the estimates.
- Outlier detection for compositional data using (robust) Mahalanobis
distances in isometric logratio coordinates is implemented in
outCoDa()function of robCompositions package.
- The compositions package provides functions to detect various types of outliers in compositional datasets.
- The kuiper.2samp package performs the two-sample Kuiper test to assess the anomaly of continuous, one-dimensional probability distributions.
enpls.od()function in enpls package performs outlier detection with ensemble partial least squares.
- The surveyoutliers package helps manage outliers in sample surveys by calculating optimal one-sided winsorizing cutoffs.
- The faoutlier package provides tools for detecting and summarize influential cases that can affect exploratory and confirmatory factor analysis models and structural equation models.
- DDoutlier (core)
- HDoutliers (core)
- OutliersO3 (core)