Skip to content

Latest commit

 

History

History
83 lines (54 loc) · 2.17 KB

13-01.md

File metadata and controls

83 lines (54 loc) · 2.17 KB
Raw
title date
Class 4
January 13, 2023

Stochastic Process ${ X(t), t\in T }$

  • $X(t) : \Omega \rightarrow \mathcal{S}$
  • $T$ and $\mathcal{S}$ are parameter and state space respectively

DTMC

  • Obeys Markov Property

Consider Transition Probability Matrix

$$ P = \begin{pmatrix} 0.9 & 0.1 \\ 0.1 & 0.9 \end{pmatrix} $$

Now if we consider $S_n = \sum X_i$ and $\hat{\mu}_n = S_n / n$ then $\hat{\mu}_n$ will depend on $P$ and might not go as expected from Strong Law of Large Numbers (SLLN).

  • $\overline{\mu}$: Initial Distribution over $\mathcal{S}$

Chapman Kolmogorov Equation for DTMC

  • This follows only for homogeneous DTMCs
    • For non-homogeneous, $P$ depends on $n$ (the timestep).

$$P^{(n + l)} = P^{(n)} P^{(l)}$$

Classification of States

Accessible State

  • $j$ is accessible from $i$ if $p^n_{ij} > 0$ for some $n$
  • Denoted as $i \rightarrow j$
  • $i \rightarrow j$ and $j \rightarrow i$ then we say that $i$ and $j$ communicate. This is denoted by $i \leftrightarrow j$.

Communication is an equivalence relation.

$i \leftrightarrow i$ achieved in $P^0 = I$.

States which communicate belong to the same equivalence class

Irreducible

Irreducible when $i \leftrightarrow j, \ \forall i, j \in \mathcal{S}$ i.e., only one communicating class.

Recurrent and Transient States

$$F_{ii} = P(\text{ever returning to } i \text{ having started in } i)$$

  • State $i$ is recurrent if $F_{ii} = 1$
  • If $F_{ii} < 1$ then State $i$ is transient
    • Each communicating class share recurrence or transience

Limiting Probabilities

The dependence on initial distribution starts to vanish after long time.

$$ \pi_j = \lim_{n\rightarrow \infty} p^{(n)}{ij} = [\lim{n \rightarrow \infty} P^n]_{ij} $$

Stationary Distribution

$$ \pi = \pi P $$ \begin{proof} Stochastic Matrices always have an eigenvalue $1$, so stationary distribution will always exist. \end{proof}

  • $\pi P$ is essentially the pmf of $X_1$ having picked $X_0$ according to $\pi$
  • $\pi = \pi P$ says that the distribution of $X_1$ will also be $\pi$ (if initial distribution was $\pi$)

\begin{remark} If Limiting Distribution exists, then it is same as stationary distribution. \end{remark}