simple_descriptive_techniques.Rmd

---
title: "Simple Exploration Techniques for Time Series Analysis"
author: "Amir R"
output:
  ioslides_presentation:
    incremental: true
    widescreen: true
    toc: true
    toc_depth: 2
    toc_float: true
    df_print: paged
---
```{r Packages, eval=T, include=FALSE}
library(magrittr) # for the pipes
# install.packages("magrittr") if not installed
```


# Intro

---

What's a time series anyway ?


*"Is it recorded or produced with temporal information ?"*

---

frequently encountered in day to day life

* Marketing: Prices, Sales, economic indices
* Central Beauro of Statistics: Demographic information
* Physical: such as Weather history

frequently encountered in research

* Observation Counts: Pop. & community ecology
* Environmental 
* recording instrument output

---

Data fit for classic time series analysis in base R

 * The Time Variable/Axis 
  
    + No Missing Values 
    + Equidistant
  
 * The Response Variable should be continuous or pseudo continuous

---

## Goals For today

1. Learn some basic R tools for exploring series

   <!-- * No packages other than the pre-installed ones -->

   <!-- * Introduce only things that can be relevant to everybody -->

2. Understand and quantify Serial correlation in time

3. Preparation for further analysis

<!-- construct a set of clues in terms of which lags are important in our system's dynamics. -->

<!-- Without going over methodology, assumptions checking, theory and math notation & formulae -->

---

## Why bother ?

* Variables in Time almost always behave differently than in space

<!-- and the areas of statistics that we usually study in the context of biology is, in many cases, not fit to handle them -->

* Explaining Observations - Multiple regression models are helpful, but they are not really designed to handle real time-series data

* For Real time series - Sample Summary Statistics (e.g.. Sample Mean and Variance )  are potentially misleading by themselves.

  + They do not have their usual properties
  + we can compute them (also for Time Series Statistics), but to describe populations we better remove systematic components from the samples first.
  
<!-- * Very developed mathematical field with many applications in  -->

<!--   + Electromagnetic / Audio Signal Analysis -->
<!--   + Signal Processing -->
<!--   + Sales / Marketing  -->
<!--   + Econometrics -->
<!--   + Materials and Manufacturing Control -->

<!-- * Much literature and expertise outside Ecology / Biology -->
<!-- = opportunity -->

---

## Objectives of Time Series Analysis

 * Description - what we are here for today
 * Explanation
 * Predictions, both with and without external variables
 * Control
 * Detection of anomalies

<!-- ## Approaches -->

<!--  * Simple Descriptive techniques - plotting, looking for cycles etc..... -->
<!--  * Inference by Analysis in the Time Domain - based on the Correlogram -->
<!--  * Inference by Analysis in the Frequency Domain - based on the Spectrogram -->
<!--  * Linear systems - one system is considered the input, another the output -->
<!--  * Modelling and Model Comparison  -->

---

Let's load some sample data and examine its structure

```{r}
blowfly <- read.csv("blowfly.csv", header = T) 
blowfly
```

---

Time for a first look at Nicholson's flies 
```{r}
plot(blowfly$total)
```

We can see some patterns already...

---

and using a time series plotting method:

```{r}
plot.ts(blowfly$total)
```

---

Multiple Time Series

```{r}
plot.ts(blowfly[1:3])
```

---

# The `ts` class 

---

`plot.ts` plots the object as if it were of the special class of R objects known simply as *ts*

the `ts()` function is used to convert objects to the ts class

```{r}
flies <- ts(blowfly$total)
eggs<-ts(blowfly$eggs)

nonemerging<- ts(blowfly$nonemerging)
emerging<- ts(blowfly$emerging)


class(flies)
typeof(flies)
```

---

setting the `start`, `end`, and `frequency` parameters and functions

```{r}
start(flies) ; end(flies) ; frequency(flies)
```

---

Add a start year with `start` or `st`
```{r}
flies<-ts(blowfly$total, start=1954)
plot(flies)
print(flies,calendar=T) 
```

---

Set time units and frequency with `start` and `frequency`
the frequency will be the number of columns

example: set a quarterly frequency
```{r}
flies<-ts(blowfly$total, start=c(1954,1),frequency = 4)
plot(flies)
print(flies,calendar=T)
```

---

set a monthly frequency
```{r}
flies<-ts(blowfly$total, start=c(1954,1), frequency = 12)
print(flies,calendar=T)
plot(flies)
```

---

explore different persepctives according to frequency with `aggregate` and `boxplot`
```{r}
par(mfrow=c(2,2))
plot(flies)
ag.flies<-aggregate(flies) ; plot(ag.flies)
cyc.flies<- cycle(flies) ; boxplot(flies ~ cyc.flies)
par(mfrow=c(1,1))
```
`cycle` was used to pull up the monthly interval as a categorical variable

---

Draw 2 series that have the same time axis together,
with `cbind` 
```{r}
eggs2<-ts(blowfly$eggs, start=c(1954,1), frequency = 12)
plot(cbind(flies , eggs2 ))
```

---

set a frequency of every 2-3 days
```{r}
flies<-ts(blowfly$total, start=(c(1954,1)) , freq = 150)
print(flies,calendar=T)
plot(flies)
```

---

```{r}
par(mfrow=c(2,1))
blowfly$eggs %>% ts(frequency= 1) %>% plot
blowfly$eggs %>% ts(frequency= 52) %>% plot
par(mfrow=c(1,1))
```

---

the 'ts' class allows R to understand

 * that a vector is based on a uni-directional time sequence
 * convert frequencies easily

many base/stats functions are actually designed for the 'ts' class

but using them on a non- *ts* object may or may not result in an error

Warning: the `ts` class is designed only for equidistant series. other packages have other classes and functions which extend capabilities

---

# Time series exploration & Descriptive techniques 

---

```{r echo=FALSE}
plot(flies)
```

   * relatively clear cyclical process, at least at the beginning (points 1:200).
   * the second part (200:361) looks less regular and with an upward trend.

---

## Exploratory Questions

 * what are the cycle periods ?
 * is there any hidden periodical behaviour that is not immediately apparent ?
 * Do these 2 parts behave similarly or is there a fundamental change ? which attributes  of the first part are conserved in the second part

```{r echo=FALSE}
plot(flies)
```

---

We can see some clearer cyclical behaviour of a section if we zoom in and explore a bit with `window`:

```{r}
flies<-ts(blowfly$total)
plot(window(flies, 1, 100, 1))
```

---

more zooming with `deltat`:

```{r}
par(mfrow=c(3,1))
plot(window(flies, 1, 100, deltat = 5))
plot(window(flies, 1, 100, deltat = 10))
plot(window(flies, 1, 100, deltat = 20))
par(mfrow=c(1,1))
```

--- 

```{r}
plot(window(flies, 1, 100, deltat = 20))
```

We finally lose the cyclical behaviour at ~`deltat=20`. let's remember this...

---

The flies series is unusually clear, and still - its cyclical behaviour is only one possible pattern.

other series may represent processes that behave in a way that is not immediately recognizable, and have many things going at once.

---

examples

```{r synthetic series, include=FALSE}
w <- rnorm(500, 0, 1) # this is actually a perfect white noise
rw <- cumsum(rnorm(100, 0, 1))
rwd <- cumsum(rnorm(200, 0, 1) + 0.05) # random walk with drift
sine.wn <- sin(1:200 / 5) + rnorm(200, mean = 20, sd = 1.5) + seq(0, 3, length.out = 200)
Z <- rnorm(250, 0, 1)
Y <- numeric(250)
Y[1] <- Z[1]
for (i in 2:250) Y[i] <- -0.5 * Y[i - 1] + Z[i]
```

```{r echo=FALSE}
plot.ts(w)
```

---

```{r echo=FALSE}
plot.ts(rwd)
```

---

```{r echo=FALSE}
plot.ts(sine.wn)
```

---

```{r echo=FALSE}
plot.ts(Y)
```

---

We naturally want  tools that will describe the entire series' and parts of the series' attributes, be executable fast in R, and guide us in further steps (such as model fitting and forecasting). 

for that, we need to understand how a given series evolves through time - via autocorrelation and related attributes of time series

---

## Measures of serial dependence 

The Lag Plot

Auto-correlation

Partial Auto-correlation

Cross-Correlation

---

### Background on Autocorrelation - The lag plot.


how does my data behave in relation to itself ? 
or 'how does this week's population relate to last week's population ?

```{r}
plot(flies[1:350], flies[2:351])
```

Clearly, there's a pattern at lag =1, at least in the smaller population sizes

---

now let's look at lags 2 and 3:

```{r fig.show='hold'}
par(mfrow=c(1,2))
plot(flies[1:350], flies[3:352])
plot(flies[1:350], flies[4:353])
par(mfrow=c(1,1))
```

we start losing the pattern gradually, as we go to higher lags...

----

more lags, lag = 1 to lag =8, using `lag.plot()`

```{r}
lag.plot(flies, lag = 8, layout = c(2, 4), diag = T)
```

---

however, at lags 9-16:

```{r}
lag.plot(flies, lags = 8, set.lags = 9:16, layout = c(2, 4), diag = T)
```

---

**summary**

 * we begin with a pattern, lose it, and then find it again
 * at smaller populations, we begin with a positive correlation, lose it, then a negative correlation  which peaks at about lag = 10, lose it, and so the cycle goes.

---

### The Correlogram, using `acf()`
```{r}
acf(flies)
plot(flies)
```

---

This summarizes and charts the autocorrelation structure in the series, up to a default lag size, which is 25 in this case.

The vertical axis is the  magnitude of the  **Sample Autocorrelation Coefficients**. they measure the relative correlation between observations at different distances apart (lags)

and the horizontal axis is simply the lag size.

---

let's have a look at these **Sample Autocorrelation Coefficients**

```{r results='hide'}
a <- acf(flies, plot = F)
```

```{r }
a$acf %>% head(10) # for the actual factors
```

---

there's no better evidence of cycles, the flies exhibit strong positive AND negative correlation at regular intervals throughout the series, with a cycle period of 19 weeks.

```{r echo = FALSE}
acf(flies)
```



---

let's look at the entire series
```{r}
acf(flies, lag.max = 360, col = "red")
```

---

We can see

* the (sort of regular) cyclical pattern, especially at the beginning
* the diminishing positive correlation
* some alternation between positive and negative correlations, which is typical of cycles.

---

```{r}
acf(flies, lag.max = 360, col = "red")
```

warning: there is bias here with increasing lag size. why ?

---

  * there's some kind of a diminishing but long term "memory" in the system, up to a lag of 100 weeks.

     + events that happened 50 weeks ago may still have an influence.
     
  * clear cycles with a period of 19 weeks

---

The main point :there's a **LOT** of non-randomness in the sequence

This last point is especially important for linear and other modelling, which assume that samples are independent. we may want to stop and think before going on to perform a conventional `lm()` ! , and at the very least be aware of series' behaviour.

and this illustrates why we need to explore time series internally before we go on to the other, external parameters.

---

The **Sample Auto-covariance Coefficients**

```{r eval=T, include=T}
acf(flies, type = "covariance")
```

---

```{r eval=T, include=T}
acf(flies, type = "covariance", plot = F)$acf %>% head(10)
```

---

### Partial Autocorrelation


Correlograms produced using the sample *autocorrelation* coefficients do not account for the fact that for a given lag size there may be correlation between internal points, (e.g.. values that are 4 points apart were correlated, but so do values that are 2 points apart).

We sometimes want to control for the internal correlations inside the lag, or in other words, to check what would have been the correlation coefficients, had all the internal lags' coefficients were forced to zero. 

This helps us later on, when selecting a model

---

the **Partial Autocorrelation ** is the relationship between this week's population and the population at lag n when we have already controlled for the correlations between all of the successive weeks between this week and week n

---

We only need to use `pacf` or `acf(type="partial")` instead of the default

```{r echo=TRUE}
acc <- a$acf %>% head(12) %>% round(2)

partial.acc <- pacf(flies, plot = F) %>% .$acf %>% head(12) %>% round(2)

acc
partial.acc
```

---

```{r}
plot(1:12, acc, xlab = "Lag", ylab = "Coefficient")
points(1:12, partial.acc, col = "blue")
lines(1:12, rep(0, 12))
legend("topright", c("ac coefficients", "Partial ac coefficients"), col = c("black", "green"), pch = c(1, 1))
```

---

```{r}
pacf(flies)
```

---

what can we take from this ?

* the maximal negative and maximal positive lags are important places in the cycle.

* clues for possible underlying processes in lags 3 and 12

the PACF can be thought of as analogous to a derivative of ACF - thus finding points driving change in pitch or direction. 

---

Summary till now, using just Correlograms for the entire series

* cycle length - about 19
* Lags 2-4, 12-16 are suspected: we should look for biological mechanisms there, when we construct an explanatory model.

* Note: these are clues generated only from the time series itself, before considering any other data or knowledge in biology

---

### Cross Correlation

The total number of flies is not the only one we have...

we could always scatter-plot the data frame variables against each other

---

```{r}
plot(blowfly, lower.panel = NULL)
```

we get a strong correlation between total, deaths, and emerging. but there's an inherent problem in this analysis...

---

```{r}

flies <- ts(blowfly$total)

ccf(nonemerging, flies)
ccf(emerging, flies)
ccf(eggs, flies)
```

---

Reminder: Lags 2-4 (negative), 12-16 (positive) are suspected from `pacf()` of the total of "flies"

from `ccf()`

example, eggs vs. total- 

* negative at lags 2-3 
* negative at (-17 - -16)
* positive at 12-13

it can be difficult to construct an explanation from this alone, but at least we now have clues of lead times and delay times between every two variables.

This is a first step in time series stats, towards modelling.

---

findings:
cycle length = still 19

one possible explanation: larval processes *appear* to drive the cycles when they reach a maximum (density dependent competition ?)

---

### Spectral Analysis
an alternative , complementary approach to analysing fluctuations is to analyse how the variance evolves through time, by distributing it to frequencies.

This method can detect important processes that may not be found with `acf() ` to to too much noise in certain lags

#### The Periodogram, using `spectrum`

horizontal axis - frequencies instead of time
vertical - variance

```{r}
spectrum(flies, main = "")
```

---

in this analysis we look for frequency peaks. in this case:

```{r}
spec<-spectrum(flies, main = "")
spec$spec %>% which.max() -> maxloc # where's the maximal value ?
max.freq <- (spec$freq)[maxloc] # what's the maximal value ?
max.freq
```

---

now let's plot and find the exact value in time units
```{r}
plot(spec, main = "")
abline(v = max.freq, col = "red", lwd = 2)
```

---

```{r}
1 / max.freq
```

suggests again cycles of ~19 years.

the information may or may not be easier to interpret than a correlogram.

it's best to use both when exploring.

---

## typical series and interpretations

```{r echo=FALSE}
par(mfrow = c(3, 1))
plot(w)
acf(w, main = "")
spectrum(w, main = "")
par(mfrow = c(1, 1))
```

No definitive pattern in lags beyond 0 or a dominant frequency

typical of a *Pure White Noise* process, which is Normal errors around a steady mean. 

---

```{r echo=FALSE}
par(mfrow = c(3, 1))
plot(ts(rw))
acf(rw, main = "")
spectrum(rw, main = "")
par(mfrow = c(1, 1))
```

No definitive pattern in lags, but high correlation for all lags

typical of a *Random Walk* process, where values are correlated to (mostly) only the latest value (the values in the previous step), plus a small *white noise* 

---

**Super Important **

Is there a real trend here ? the time plot and our brain say there is. but the underlying process is probably not a *real* trend or combination of trends, but a kind of random 'drift'.

the next value is more likely to be a minor deviation from the last value, with equal odds for moving up or down, than a result of a another external driver ! 

random walks are notorious in deceiving our brains.
actually, the long term mean of the formula used to create this series,
`rw<-cumsum(rnorm(100,0,1))`
**is 0 !**

---

repeat:

```{r echo=FALSE}
rw <- cumsum(rnorm(100, 0, 1))
par(mfrow = c(3, 1))
plot(ts(rw))
acf(rw, main = "")
spectrum(rw, main = "")
par(mfrow = c(1, 1))
```

The same ACF even though the time plot is completely different

---

```{r echo=FALSE}
par(mfrow = c(3, 1))
plot(Y)
acf(Y)
pacf(Y, main = "")
par(mfrow = c(1, 1))
```

---

repeated negative followed by positive:
return to equilibrium following random departures from it.

This is called an **Autoregressive Process** of order 1 or 'AR(1)' (the order can be determined from the `pacf()`)
in this case a *negative* AR(1) with rapid response.

ecological models of equilibrium population dynamics are typically Autoregressive.

---

## analysing parts of a TS + de-trending


Simplest Approach:

1. visually identify a breakpoint

2. apply the same steps on parts of the series to find their attributes and identify discontinuities

---

now let's adapt the analysis for the two parts, separately


split the series:
```{r, fig.show='hold'}
first <- ts(flies[1:200])
second <- ts(flies[201:361])
```

---

```{r echo=FALSE, fig.show='hold'}
par(mfrow = c(1, 2))
plot(first, ylim = c(0, 14000))
plot(second)
par(mfrow = c(1, 1))
```

what are the notable differences between the 2 parts ?
<!-- e.g same cycle periods ? more 'condensed' ? -->

---

```{r acf on both parts, echo=FALSE, fig.show="hold"}
par(mfrow = c(2, 2))
acf(first)
acf(second)
pacf(first)
pacf(second)
par(mfrow = c(1, 1))
```

same overall shapes, same cycle length

---

Important Lags in the *second* parts are shifted 1 or 2, compared with the *first* - change points are somewhat closer to each other

what if we wanted to analyse the two with a commom baseline ?
<!-- the first and second parts appear to follow different trends -->

---

### de-trending using `lm()`

let's de-trend using linear regression, and have look at the results.

```{r de-trending usin a LM, results='hold'}
firstX <- 1:length(first)
first.lm <- lm(first ~ firstX)
first.detrended <- first - predict(first.lm)

secondX <- 1:length(second)
second.lm <- lm(second ~ secondX)
second.detrended <- second - predict(second.lm)
```

---

```{r de-trending using a LM, results='hold'}
par(mfrow = c(2, 2))
plot(first) ; lines(first.lm$fitted.values, col = "red")
plot(second) ; lines(second.lm$fitted.values, col = "blue")
plot.ts(first.detrended) # reminder Q: why plot.ts ?
plot.ts(second.detrended)
par(mfrow = c(1, 1))
```

---

```{r}
first.lm$coefficients
second.lm$coefficients
```

note: the pitch of of 'second' is 5 times that of first

---

now, let's compare again

```{r acf on both parts, detrended, fig.show="hold"}
par(mfrow = c(2, 2))
acf(first.detrended) ; acf(second.detrended)
pacf(first.detrended); pacf(second.detrended)
par(mfrow = c(1, 1))
```

---

comparing the de-trended versions:
`acf()` - results similar, but `pacf()` are quite different

conclusions:

* suspected trend change
* continuity in cycle period at the same time as a trend change, 
* quite a difference in the underlying change points
* also: a new negative partial coefficient at lag 18

---

### de-trending using `diff()`

```{r}
par(mfrow = c(2, 1))
plot(flies)
flies %>% diff() %>% plot()
par(mfrow = c(1, 1))
```

---

```{r echo=FALSE}
first.diff <- diff(first)
second.diff <- diff(second)


par(mfrow = c(3, 2))
plot(first)
lines(first.lm$fitted.values, col = "red")
plot(second)
lines(second.lm$fitted.values, col = "blue")

plot.ts(first.detrended) # reminder Q: why plot.ts ?
plot.ts(second.detrended)

plot(first.diff)
plot(second.diff)

par(mfrow = c(1, 1))
```

```{r echo=FALSE}
par(mfrow = c(3, 2))

plot(first)
plot(second)

acf(first.detrended, main = "")
acf(second.detrended, main = "")

acf(first.diff, main = "")
acf(second.diff, main = "")
par(mfrow = c(1, 1))
```

### De-trending Remarks

De-trending is the goal of many methods in time series analysis that are not presented here


+first order differencing will usual be enough coerce the series to become stationary, and will do it while still keeping the attributes of the signal. de-trending by a `lm()` may create noise that was not there.

it has several uses in model selection, and the advantage that parameters do not need to be estimated (e.g. from a linear model)


***

## Smoothing

let's return to some noisier series

```{r include=FALSE}
w <- ts(w)
rwd <- ts(rwd)
sine.wn <- ts(sine.wn)
Y <- ts(Y)
```

```{r}
par(mfrow=c(2,2))
plot(w)
plot(rwd)
plot(sine.wn)
plot(Y)
par(mfrow=c(1,1))
```

---

last time we used `window()`

```{r}
par(mfrow = c(2, 2))

plot(window(w, 100, 200))
plot(window(w, 100, 150))
plot(window(w, 100, 125))

par(mfrow = c(1, 1))
```

this approach doesn't work

we should probably start looking by smoothing first

### smoothing via `filter()`

`filter` allows many methods for smoothing

#### filtering with a moving average

this is the simplest technique
we use the filter argument with a weights vector, giving equal weights

```{r, fig.show="hold"}
par(mfrow = c(3, 1))

v <- filter(w, sides = 2, filter = rep(1, 3) / 3)
plot(w)
lines(v, col = "red")

rwd.s <- filter(rwd, sides = 2, filter = rep(1, 3) / 3)
plot(rwd)
lines(rwd.s, col = "red")

sine.wn.s <- filter(sine.wn, sides = 2, filter = rep(1, 3) / 3)
plot(sine.wn)
lines(sine.wn.s, col = "red")

Y.s <- filter(Y, sides = 2, filter = rep(1, 3) / 3)
plot(Y)
lines(Y.s, col = "red")

par(mfrow = c(1, 1))
```


let's try a 9 point average

```{r,fig.show="hold"}
v <- filter(w, sides = 2, filter = rep(1, 9) / 9)
plot(w)
lines(v, col = "red")

rwd.s <- filter(rwd, sides = 2, filter = rep(1, 9) / 9)
plot(rwd)
lines(rwd.s, col = "red")

sine.wn.s <- filter(sine.wn, sides = 2, filter = rep(1, 9) / 9)
plot(sine.wn)
lines(sine.wn.s, col = "red")


Y.s <- filter(Y, sides = 2, filter = rep(1, 9) / 9)
plot(Y)
lines(Y.s, col = "red")
```

let's try a 15 point average

```{r,fig.show="hold"}
v <- filter(w, sides = 2, filter = rep(1, 15) / 15)
plot(w)
lines(v, col = "red")

rwd.s <- filter(rwd, sides = 2, filter = rep(1, 15) / 15)
plot(rwd)
lines(rwd.s, col = "red")

sine.wn.s <- filter(sine.wn, sides = 2, filter = rep(1, 15) / 15)
plot(sine.wn)
lines(sine.wn.s, col = "red", lwd = 3)


Y.s <- filter(Y, sides = 2, filter = rep(1, 15) / 15)
plot(Y)
lines(Y.s, col = "red")
```

increase the averaging period to receive a smoother results and (maybe) reveal patterns in the noise


a moving average is a *local smoother*, meaning it allows a change of series behaviour through out the series.

3### smoothing via `ksmooth()`

```{r}
t <- 1:length(sine.wn)

plot(sine.wn)
lines(sine.wn.s, col = "red", lwd = 3)
lines(ksmooth(t, sine.wn, "normal", bandwidth = 5), col = "blue", lwd = 2)

plot(sine.wn)
lines(sine.wn.s, col = "red", lwd = 3)
lines(ksmooth(t, sine.wn, "normal", bandwidth = 10), col = "blue", lwd = 2)

plot(sine.wn)
lines(sine.wn.s, col = "red", lwd = 3)
lines(ksmooth(t, sine.wn, "normal", bandwidth = 20), col = "blue", lwd = 2)
```

The wider the bandwidth, the smoother the result

#### smoothing via `smooth.spline()`

```{r}
t <- 1:length(sine.wn)

# smoothing paraeter=0.1
plot(sine.wn)
lines(sine.wn.s, col = "red", lwd = 3)
lines(smooth.spline(t, sine.wn, spar = 0.1), col = "blue", lwd = 2)

# smoothing paraeter=1
plot(sine.wn)
lines(sine.wn.s, col = "red", lwd = 3)
lines(smooth.spline(t, sine.wn, spar = 1), col = "blue", lwd = 2)

# smoothing paraeter=0.7
plot(sine.wn)
lines(sine.wn.s, col = "red", lwd = 3)
lines(smooth.spline(t, sine.wn, spar = 0.7), col = "blue", lwd = 2)
```

#### smoothing via several methods

```{r}
t <- 1:length(flies)

plot(flies)
lines(ksmooth(t, flies, "normal", bandwidth = 5), col = "blue", lwd = 2)

# spline smoothing smoothing paraeter=0.4
plot(flies)
lines(smooth.spline(t, flies, spar = 0.4), col = "blue", lwd = 2)
```

***

# Conclusion & Remarks

---

The description of a given time series is usually a cyclical process by itself

some methods that are covered here are also useful in other contexts (e,g. model fitting, control)

---

## Further Steps

* Model fitting or Comparison
* Forecasting 
* Discussion of outliers
* Statistical Theory
* Statistical Methodology

---

## General Warnings

 * many methods, not described here, do have some underlying assumption. most commonly the assumption of *Stationarity* 
 * Most of the methodology of time series covers **only discrete time and continuous (or pseudo-continuous) variable** (e.g. binary and point process are not covered)
 * many of the methods do not describe short and relatively stable series ( like a 10 points richness series) well enough for analysis

---

# Terminology & Abbreviations 



## R functions {.smaller}
| function | Purpose |
|:---------|:--------|
| ts       | construct a time series object |
| ts.plot  | plot a vector or matrix as if they were of the special 'ts' class.
| window   | have a look at sections of series |
| lag.plot | create a scatter-plot matrix by lags |
| acf , pacf , ccf | study serial correlation
| Diff | recode the series by differencing |
| lm | construct a Linear Model, such a simple univariate regression to remove a trend |
| filter, ksmooth, smooth.spline | applying linear and non linear filters. in the context of this lesson - for smoothing |

## Process Terminology


*Process* - a mathematical description of observations as a function of time (and perhaps other variables)

*Stationary Process* (intuitive definition, non formal) : 

 * No systemic change in mean (no trend)
 * No systemic change in variance
 * Strictly periodic variations have been removed

---

In other words, The properties of various sections of the data are constant

 * Usually in the context of model fitting
 * Non Stationarity components (trend, cycles) may be the focus, or we may want to remove them prior to analysis
 * much of the probability theory of time series analysis concerns only stationary series, so we may need to transform a given series to stationary

NOTE: No real process is strictly Stationary !

---

## Series Terminology {.smaller}

| Term | Refers to |
|:-----|:----------| 
| Continuous | Observations are made continuously through time (even if the variable of interest can only take certain) |
| Deterministic | a time series that can be predicted exactly, by a well defined mathematical process |
| Discrete | observations are taken at specific times |
| Equidistant  | A series where the variable is recorded at equal intervals (e.g every month) |
| Lag | The time- distance between points in the series, also used for delay lengths in process/series descriptions |
| Stationary | See above |
| Stochastic | The future is only partly determined by the past/ exact predictions are impossible and are replaced by the idea that future values have a probability distribution |


## Process Types Terminology {.smaller}

| Term | Refers to |
|:-----|:----------|   
| Point Process    | Binary Observations occur randomly at non equal interval |    
| Binary Process   | Observations can take only "0" or "1", Yes/No and so on


## Abbreviations and Notation

| Term        | Refers to |
|:------------|:----------| 
| ACF or ac.f | Autocorrelation Function |  
| pACF | Partial (corr. coefficient) ACF |  
| AR   | Autoregressive |  
| MA   | Moving Average |  


# Source Material and useful links

## Useful Links

CRAN Task View: Time Series Analysis
https://cran.r-project.org/web/views/TimeSeries.html

Time Series Modelling basics, explained simply albeit with some theory mistakes
https://www.analyticsvidhya.com/blog/2015/12/complete-tutorial-time-series-modelling/

Basic Data Analysis for Time Series with R
https://library.books24x7.com/toc.aspx?bookid=63461
(requires a library login + a books 24/7 account)

---

Looking for real ecological time series ?

* BIOTIME - a Global database of biodiversity time series
http://biotime.st-andrews.ac.uk/

* NEON - a repository for LTERs

https://data.neonscience.org/browse-data?showAllDates=true&showAllSites=true&showTheme=org

* DEIMS - another repository for LTERs
https://deims.org/

## Bibliography

Heavily based on some Textbooks and R manuals
 
 + Chatfield - Chapters 1, 2 & 6
 Chatfield, Christopher. 2004. The Analysis of Time Series: An Introduction. 6th ed. Texts in Statistical Science. Boca Raton, FL: Chapman & Hall/CRC.
 + Shumway & Stoffer - Chapters 1, 2 & 4
 Shumway, Robert H., and David S. Stoffer. 2006. Time Series Analysis and Its Applications: With R Examples. 2nd [updated] ed. Springer Texts in Statistics. New York: Springer.
 + The R book by Crawley - Chapter 24
Crawley, Michael J. 2013. The R Book. Second edition. Chichester, West Sussex, United Kingdom: Wiley.

There are many others...