notebooks/adaptive_filters_bare_minimum.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adaptive Filters and Neural Units - Bare Minimum of Informations\n",
    "## Introduction\n",
    "\n",
    "An adaptive filter is a system with a linear filter that has a transfer function controlled by variable parameters and a means to adjust those parameters according to an optimization algorithm.\n",
    "\n",
    "The output of adaptive filter $y(k)$ is produced as follows\n",
    "\n",
    "$y(k) = w_1 x_{1}(k) + ... + w_n x_{n}(k)$,\n",
    "\n",
    "or in a vector form\n",
    "\n",
    "$y(k) = \\textbf{x}^T(k) \\textbf{w}(k)$,\n",
    "\n",
    "where $k$ is discrete time index, $(.)^{T}$ denotes the transposition, $y(k)$ is filtered signal, $\\textbf{w}$ is vector of filter adaptive parameters and $\\textbf{x}$ is input vector (for a filter of size $n$) as follows\n",
    "\n",
    "$\\textbf{x}(k) = [x_1(k), ...,  x_n(k)]$.\n",
    "\n",
    "### Initial Values of Adapative Parameters\n",
    "\n",
    "The values of adaptive weights (adaptive parameters) $\\textbf{w}$ are usually set to all zeros, or alternatively to random numbers (normal distribution, zero mean value).\n",
    "\n",
    "### Augmentation with Bias\n",
    "\n",
    "The input vector $\\textbf{x}(k)$ can be extended with bias. Common used value for bias is $1$. The augmented input vector can look as follows\n",
    "\n",
    "$\\textbf{x}(k) = [1, x_1(k), ...,  x_n(k)]$.\n",
    "\n",
    "This change also means that vector of adaptive parameters have to change - it should be extended by one item to match the size of input vector. The bias do not change during the adaptation process.\n",
    "\n",
    "In some fields (computational intelligence etc.) this augmented version seems to be prefered. The adaptive filter with this modification is often called Neural Unit.\n",
    "\n",
    "### Terminology in different fields\n",
    "\n",
    "* Adaptive Filter = Neural Unit (NU are most likely adaptive filters with random initial adaptive parameters and with bias)\n",
    "\n",
    "* desired value = target\n",
    "\n",
    "* least-mean-squares = stochastic gradiend descent\n",
    "\n",
    "## Learning Algorithms\n",
    "\n",
    "The most popular learning algorithms for adaptive filters are:\n",
    "\n",
    "* **Least-mean-squares** (LMS) (this is implementation of Stochastic Gradiend Descent), commonly is also noted just as Gradiend Descent adaptation\n",
    "\n",
    "* **Normalized Least-mean-squares** (NLMS) - this is modification of LMS\n",
    "\n",
    "* **Recursive Least Squares** (RLS) \n",
    "\n",
    "It is hard to say, what algorithms produce the best results. The performance is strongly task dependand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}