-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
12 changed files
with
198 additions
and
41 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,7 +2,7 @@ language: julia | |
|
||
julia: | ||
- 1.0 | ||
- 1.2 | ||
- 1.3 | ||
|
||
os: | ||
- linux | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
# Exploration Policies | ||
|
||
Exploration policies are often useful for Reinforcement Learning algorithm to choose an action that is different than the action given by the policy being learned. | ||
|
||
This package provides two exploration policies: `EpsGreedyPolicy` and `SoftmaxPolicy` | ||
|
||
```@docs | ||
EpsGreedyPolicy | ||
SoftmaxPolicy | ||
``` | ||
|
||
## Interface | ||
|
||
Exploration policies are subtype of the abstract `ExplorationPolicy` type and they follow the following interface: | ||
`action(exploration_policy::ExplorationPolicy, on_policy::Policy, s)`. | ||
|
||
The `action` method is exported by [POMDPs.jl](https://github.com/JuliaPOMDP/POMDPs.jl). | ||
To use exploration policies in a solver, you must use the three argument version of `action` where `on_policy` is the policy being learned (e.g. tabular policy or neural network policy). | ||
|
||
## Schedules | ||
|
||
Exploration policies often relies on a key parameter: $\epsilon$ in $\epsilon$-greedy and the temperature in softmax for example. | ||
Reinforcement learning algorithms often require a decay schedule for these parameters. | ||
`POMDPPolicies.jl` exports an interface for implementing decay schedules as well as a few convenient schedule. | ||
|
||
```@docs | ||
LinearDecaySchedule | ||
ConstantSchedule | ||
``` | ||
|
||
To implement your own schedule, you must define a schedule type that is a subtype of `ExplorationSchedule`, as well as the function `update_value` that returns the new parameter value updated according to your schedule. | ||
|
||
```@docs | ||
ExplorationSchedule | ||
update_value | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,120 @@ | ||
|
||
|
||
# exploration schedule | ||
""" | ||
ExplorationSchedule | ||
Abstract type for exploration schedule. | ||
It is useful to define the schedule of a parameter of an exploration policy. | ||
The effect of a schedule is defined by the `update_value` function. | ||
""" | ||
abstract type ExplorationSchedule end | ||
|
||
""" | ||
update_value(::ExplorationSchedule, value) | ||
Returns an updated value according to the schedule. | ||
""" | ||
function update_value(::ExplorationSchedule, value) end | ||
|
||
|
||
""" | ||
LinearDecaySchedule | ||
A schedule that linearly decreases a value from `start_val` to `end_val` in `steps` steps. | ||
if the value is greater or equal to `end_val`, it stays constant. | ||
# Constructor | ||
`LinearDecaySchedule(;start_val, end_val, steps)` | ||
""" | ||
@with_kw struct LinearDecaySchedule{R<:Real} <: ExplorationSchedule | ||
start_val::R | ||
end_val::R | ||
steps::Int | ||
end | ||
|
||
function update_value(schedule::LinearDecaySchedule, value) | ||
rate = (schedule.start_val - schedule.end_val) / schedule.steps | ||
new_value = max(value - rate, schedule.end_val) | ||
end | ||
|
||
""" | ||
ConstantSchedule | ||
A schedule that keeps the value constant | ||
""" | ||
struct ConstantSchedule <: ExplorationSchedule | ||
end | ||
|
||
update_value(::ConstantSchedule, value) = value | ||
|
||
|
||
""" | ||
ExplorationPolicy <: Policy | ||
An abstract type for exploration policies. | ||
Sampling from an exploration policy is done using `action(exploration_policy, on_policy, state)` | ||
""" | ||
abstract type ExplorationPolicy <: Policy end | ||
|
||
|
||
|
||
""" | ||
EpsGreedyPolicy <: ExplorationPolicy | ||
represents an epsilon greedy policy, sampling a random action with a probability `eps` or returning an action from a given policy otherwise. | ||
The evolution of epsilon can be controlled using a schedule. This feature is useful for using those policies in reinforcement learning algorithms. | ||
constructor: | ||
`EpsGreedyPolicy(problem::Union{MDP, POMDP}, eps::Float64; rng=Random.GLOBAL_RNG, schedule=ConstantSchedule)` | ||
""" | ||
mutable struct EpsGreedyPolicy{T<:Real, S<:ExplorationSchedule, R<:AbstractRNG, A} <: ExplorationPolicy | ||
eps::T | ||
schedule::S | ||
rng::R | ||
actions::A | ||
end | ||
|
||
function EpsGreedyPolicy(problem::Union{MDP, POMDP}, eps::Real; | ||
rng::AbstractRNG=Random.GLOBAL_RNG, | ||
schedule::ExplorationSchedule=ConstantSchedule()) | ||
return EpsGreedyPolicy(eps, schedule, rng, actions(problem)) | ||
end | ||
|
||
|
||
function POMDPs.action(p::EpsGreedyPolicy{T}, on_policy::Policy, s) where T<:Real | ||
p.eps = update_value(p.schedule, p.eps) | ||
if rand(p.rng) < p.eps | ||
return rand(p.rng, p.actions) | ||
else | ||
return action(on_policy, s) | ||
end | ||
end | ||
|
||
# softmax | ||
""" | ||
SoftmaxPolicy <: ExplorationPolicy | ||
represents a softmax policy, sampling a random action according to a softmax function. | ||
The softmax function converts the action values of the on policy into probabilities that are used for sampling. | ||
A temperature parameter can be used to make the resulting distribution more or less wide. | ||
""" | ||
mutable struct SoftmaxPolicy{T<:Real, S<:ExplorationSchedule, R<:AbstractRNG, A} <: ExplorationPolicy | ||
temperature::T | ||
schedule::S | ||
rng::R | ||
actions::A | ||
end | ||
|
||
function SoftmaxPolicy(problem, temperature::Real; | ||
rng::AbstractRNG=Random.GLOBAL_RNG, | ||
schedule::ExplorationSchedule=ConstantSchedule()) | ||
return SoftmaxPolicy(temperature, schedule, rng, actions(problem)) | ||
end | ||
|
||
function POMDPs.action(p::SoftmaxPolicy, on_policy::Policy, s) | ||
p.temperature = update_value(p.schedule, p.temperature) | ||
vals = actionvalues(on_policy, s) | ||
vals ./= p.temperature | ||
maxval = maximum(vals) | ||
exp_vals = exp.(vals .- maxval) | ||
exp_vals /= sum(exp_vals) | ||
return p.actions[sample(p.rng, Weights(exp_vals))] | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
using POMDPModels | ||
|
||
problem = SimpleGridWorld() | ||
# e greedy | ||
policy = EpsGreedyPolicy(problem, 0.5) | ||
a = first(actions(problem)) | ||
@inferred action(policy, FunctionPolicy(s->a::Symbol), GWPos(1,1)) | ||
policy.eps = 0.0 | ||
@test action(policy, FunctionPolicy(s->a), GWPos(1,1)) == a | ||
|
||
# softmax | ||
policy = SoftmaxPolicy(problem, 0.5) | ||
on_policy = ValuePolicy(problem) | ||
@inferred action(policy, on_policy, GWPos(1,1)) | ||
|
||
# test linear schedule | ||
policy = EpsGreedyPolicy(problem, 1.0, schedule=LinearDecaySchedule(start_val=1.0, end_val=0.0, steps=10)) | ||
for i=1:11 | ||
action(policy, FunctionPolicy(s->a), GWPos(1,1)) | ||
@test policy.eps < 1.0 | ||
end | ||
@test policy.eps ≈ 0.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters