Interpolation for Missing Data

An overview & demo of various Interpolation methods for missing data.

Effective for Dynamic-detection(LIDAR/CFAR), Deep Image Sensor, and Time-series records with NaN entries.

Imputational trace with classical interpolation (Linear/Polynomial/Spline/Seasonal) and Machine Learning Methods.

Contents

Click Download to View, HTML

• [Basic] - Linear Interpolation
• [Advanced] - Curved Interpolation:
  • Polynomial Interpolation
  • Spline Interpolation
• Dealing with Seasonal Data:
  • Seasonal Decomposition of Time Series
  • Filling Missing Values Using Seasonal Decomposition
• Machine Learning for Imputation:
  • Using K-Nearest Neighbors to Impute Missing Values
• Notes for Data Engineers
• Demo & Comparison
  • Interpolation for Missing Data - Detailed Demo (click Download to View)

[Basic] - Linear Interpolation

Linear interpolation is a simple and effective method for estimating missing values when the data is relatively smooth, and exhibits linear trends between data points.

[Advanced] - Curved Interpolation:

• Polynomial Interpolation

  Polynomial interpolation fits a polynomial of specified order, or complex curves through the known data points, making it suitable for datasets with non-linear trends.

• Spline Interpolation

  Spline interpolation fits piecewise polynomials between data points, ensuring smooth transitions. It effectively captures complex patterns without the risk of overfitting.

Dealing with Seasonal Data:

• Seasonal Decomposition of Time Series

• Filling Missing Values Using Seasonal Decomposition

  Seasonal decomposition is highly effective for time series data with clear seasonal patterns. It separates the data into trend, seasonal, and residual components, allowing for targeted imputation. Filling missing values using the trend component can be effective in seasonal data.

Machine Learning for Imputation:

• Using K-Nearest Neighbors to Impute Missing Values

  KNN imputation is a versatile method that estimates missing values based on the similarity to other data points. It can capture local patterns in the data.

Notes for Data Engineers

Click Download to View, HTML

Demo & Comparison

• Download to View:

Interpolation for Missing Data - Demo (HTML, 3.3MB) Comparison of 6 interpolation models with dataset and illustrations.

Python solution - Linear Interpolation (.py)

Python solution - Polynomial Interpolation (.py)

Prepared & Published by:

Sun CHUNG, SMIEEE M.Sc. HKU - colab w/ MIT-IDSS

---