README.Rmd

---
always_allow_html: true
bibliography: ./vignettes/litt.bib
output: 
   github_document
editor_options: 
  chunk_output_type: console
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, include=FALSE}
library(knitr)
rgl::setupKnitr()
opts_chunk$set(
  collapse = TRUE,
  #comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "99%", fig.width = 8, fig.align = "center", fig.asp = 0.62,
  echo = TRUE, warning=FALSE, message=FALSE, 
  include = TRUE
)
library(MDP2)
## Use temp dir
wd <- setwd(tempdir())
```

<!-- badges: start -->
[![R-CMD-check](https://github.com/relund/mdp/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/relund/mdp/actions/workflows/R-CMD-check.yaml)
[![Codecov test coverage](https://codecov.io/gh/relund/mdp/branch/master/graph/badge.svg)](https://app.codecov.io/gh/relund/mdp?branch=master)
<!-- badges: end -->

# Markov Decision Processes (MDPs) in R

The `MDP2` package in R is a package for solving Markov decision processes (MDPs) with discrete 
time-steps, states and actions. Both traditional MDPs [@Puterman94], semi-Markov decision processes
(semi-MDPs) [@Tijms03] and hierarchical-MDPs (HMDPs) [@Kristensen00] can be solved under a finite
and infinite time-horizon.

Building and solving an MDP is done in two steps. First, the MDP is built and saved in a set 
of binary files. Next, you load the MDP into memory from the binary files and apply various
algorithms to the model.

The package implement well-known algorithms such as policy iteration and value iteration
under different criteria e.g. average reward per time unit and expected total discounted reward. The
model is stored using an underlying data structure based on the *state-expanded directed hypergraph*
of the MDP (@Relund06) implemented in `C++` for fast running times. 

## Installation

Install the latest stable release from CRAN:

```{r, eval=FALSE}
install.packages("MDP2")
```

Alternatively, install the latest development version from GitHub (recommended):

```{r, eval=FALSE}
remotes::install_github("relund/mdp")
```

We load the package using 

```{r Load package, echo=TRUE, warning=FALSE, results='hide', message=FALSE}
library(MDP2)
```

Help about the package can be seen by writing

```{r Package help, echo=TRUE, eval=FALSE}
?MDP2
```

To illustrate the package capabilities, we use a few examples, namely, an infinite and
finite-horizon semi-MDP and a HMDP. Before each example a short introduction to these models are
given.

## Learning more

To get started, first read `vignette("MDP2")`. 

For more examples see `example("MDP2")`. 

```{r, include=FALSE}
## reset working dir
setwd(wd)
```


## References